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ABSTRACT OF FINDINGS 



Job Corps stands out as the nation’s largest, most comprehensive education and job training 
program for disadvantaged youths. It serves disadvantaged youths between the ages of 16 and 
24, primarily in a residential setting. It provides comprehensive services — basic education, 
vocational skills training, health care and education, counseling, and residential support. Each 
year. Job Corps serves more than 60,000 new participants in about 120 centers nationwide, at a 
cost of about $1.5 billion. 

The National Job Corps Study has been conducted since 1993 under contract with the U.S. 
Department of Labor (DOE). It is intended to provide Congress and program managers with the 
information they need to assess how well Job Corps attains its goal of helping students become 
more responsible, employable, and productive citizens. 

The cornerstone of the National Job Corps Study was the random assignment of all youths 
found eligible for Job Corps to either a program group (who could enroll in Job Corps) or a 
control group (who could not). Random assignment took place between late 1994 and early 
1996. In previous reports, we presented impact and benefit-cost estimates by comparing the 
experiences — and in particular, the earnings — of the program and control groups using data from 
follow-up interviews conducted during the four years after random assignment. 

This report presents findings from an analysis of administrative earnings records. These data 
allow us to address two questions: (1) Do survey and administrative earnings data yield similar 
impact estimates on earnings during the periods covered by both data sources? and (2) What are 
estimated impacts on earnings in the two and a half years beyond the four- year period covered by 
the survey? Two sources of administrative data were collected for the study: (1) annual social 
security earnings (SER) data reported by employers to the Internal Revenue Service (IRS), and 
(2) quarterly wage records reported by employers to state unemployment insurance (UI) agencies 
in 22 randomly selected states. The SER and UI data cover nearly all workers in formal jobs. Our 
findings using these data are summarized below. 

The pattern of the estimated impacts using the survey and administrative data are similar 
in periods covered by both data sources. According to both the survey and administrative 
records data, the estimated earnings impacts are negative in the first and second years after 
random assignment (when the program group was enrolled in Job Corps) and positive and 
statistically significant in the third and fourth years after random assignment. 

However, the survey-based impact estimates are larger and more often statistically 
significant. Two factors account for the larger survey-based impacts. Eirst, reported earnings 
levels are much higher according to the survey data, due in part to social security numbers that 
may have been incorrectly reported by employers or sample members, the noncoverage of 
informal and some formal jobs in the administrative records data, and the likely overreporting of 
hours worked in the survey data. Second, according to the administrative records data, earnings 
impacts are larger for survey respondents than nonrespondents, suggesting that the survey-based 
impact estimates are slightly biased upward. 
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Based on the administrative data, we find no impacts of Job Corps for the full sample on 
employment or earnings after the four-year period covered by the survey. The estimated 
impacts in years 5 to 7 after random assignment are all near zero, and none are statistically 
significant. 

The earnings impacts for 20- to 24-year-olds at program application appear to have 
persisted. We find no beneficial earnings impacts in the post-survey period that are statistically 
significant for any subgroup. However, positive earnings gains for those 20 to 24 and those with 
a high school credential at program application persisted with little decay. 

The revised benefit-cost estimates suggest that the benefits to society of Job Corps are 
smaller than the substantial program costs. For the initial benefit-cost analysis based on the 
survey data only, we assumed that the survey-based impact estimates found in the fourth year 
after random assignment would persist without decline. This assumption generates program 
benefits that exceed program costs from society’s perspective. The administrative -based impact 
findings, however, do not support the assumption that the earnings impacts will persist without 
decline or with only modest decline after the survey observation period. The revised benefit-cost 
analysis, which assumed a higher decay rate in the earnings impacts, produced substantially 
smaller estimates of program benefits. These estimated benefits to society are lower than 
program costs. The revised benefit-cost analysis, however, finds that Job Corps benefits exceed 
costs from the perspective of participants, suggesting that the program has important 
distributional effects. 

Job Corps, however, may be cost-effective for the 20- to 24-year-olds at program 
application whose earnings impacts persisted during the post-survey period. The findings for 
the older youth can help guide future program improvement. Job Corps appears to have a 
longer-term beneficial effect on the earnings of older students than younger ones (who had 
temporary earnings gains only). Older students remain in Job Corps longer than younger ones, 
receive more hours of vocational training while enrolled, and are more highly motivated and 
well-behaved (as reported by program staff). Furthermore, many of the youngest sample 
members in the control group returned to high school after being rejected from Job Corps, 
whereas fewer older control group members enrolled in alternative education and training 
programs. These findings suggest that to improve overall program effectiveness. Job Corps needs 
to fully address differences by age in program structure and experience, and perhaps, to reassess 
the target population served by the program. 

Only further long-term followup would eliminate all uncertainties about the effectiveness 
of the program. Intensive, costly programs like Job Corps can only be expected to show benefits 
that exceed costs over a relatively long time horizon. Unfortunately, the foundation of empirical 
evidence to make long-term extrapolations of the profile of earnings in response to programs 
such as Job Corps simply does not exist. We have observed impacts for only seven years after 
program application. Consequently, we cannot discount the possibility that positive earnings 
impacts might re-emerge. Job Corps may increase the long-term earnings of those who were 16 
to 19 years old at program application as they mature, find stable jobs, and experience the full 
benefits of program participation, both from increased vocational and academic skills gained in 
the program, as well as from improved social skills and attitudes toward work. The persistent 
earnings impacts among students who were 20 to 24 at program application raise this as a 
possibility. Only continued followup of the study sample can answer this question. 



XVI 




EXECUTIVE SUMMARY 



Job Corps stands out as the nation’s largest, most comprehensive education and job training 
program for disadvantaged youths. It serves disadvantaged youths between the ages of 16 and 
24, primarily in a residential setting. It provides comprehensive services — basic education, 
vocational skills training, health care and education, counseling, and residential support. Each 
year. Job Corps serves more than 60,000 new participants in about 120 centers nationwide, at a 
cost of about $1.5 billion. 

The National Job Corps Study has been conducted since 1993 under contract with the U.S. 
Department of Labor (DOE). It is intended to provide Congress and program managers with the 
information they need to assess how well Job Corps attains its goal of helping students become 
more responsible, employable, and productive citizens. 

The cornerstone of the National Job Corps Study was the random assignment of all youths 
found eligible for Job Corps to either a program group (who could enroll in Job Corps) or a 
control group (who could not). The research sample consists of approximately 9,400 program 
group members and 6,000 control group members randomly selected from among nearly 81,000 
applicants nationwide. Random assignment took place between late 1994 and early 1996. The 
survey data for the evaluation come from interviews conducted at baseline (shortly after random 
assignment), and at 12, 30, and 48 months after random assignment. The response rate to the 
48-month interview was about 80 percent (81 percent for the program group and 78 percent for 
the control group). Program impacts were estimated by comparing the mean outcomes of 
program and control group members. 

The survey data indicate that Job Corps generated positive impacts on earnings — the key 
outcome for the study — beginning in the third year after random assignment, and the impacts 
persisted without decline through the end of the four- year follow-up period. Beneficial program 
impacts were found broadly across youth subgroups. A benefit-cost analysis based on impact 
estimates from the survey data found that the benefits to society from the program exceed its 
costs. However, this finding requires a key assumption — that the earnings gains observed during 
the last year of the observation period will persist with little decay. 

This report presents findings from an analysis of administrative earnings records. These 
data allow us to address two questions: 

1. Do survey and administrative earnings data yield similar impact estimates on 
employment and earnings during the periods covered by both data sources? 

2. What are estimated impacts on earnings and employment in the two and a half 
years beyond the four-year period covered by the survey? 



Two sources of administrative data were collected for the study: (1) annual social security 
earnings (SER) data reported by employers to the Internal Revenue Service (IRS) and Social 
Security Administration (SSA), and (2) quarterly wage records reported by employers to state 




unemployment insurance (UI) agencies in 22 randomly selected states. The SER data cover 
calendar years 1993 to 2001. The years 1995 to 1998 pertain roughly to the four- year period 
covered by the survey, and the years 1999 to 2001 pertain to the post-survey period (that is, years 
5, 6, and 7 after random assignment). The UI data cover the 1999 to 2001 period only. The SER 
and UI data cover nearly all workers in formal jobs. Earnings from informal jobs are not covered. 

IMPACT FINDINGS DURING THE PERIOD COVERED BY THE SURVEY 

The pattern of the estimated impacts using the survey and administrative data are 
similar in periods covered by both data sources. According to both the survey and SER data, 
the estimated earnings impacts are negative in 1995 and 1996 (when the program group was 
enrolled in Job Corps) and positive and statistically significant in 1997 and 1998 (Table 1). 

However, the survey-based impact estimates are larger and more often statistically 
significant. Reported earnings levels are much higher according to the survey data for a large 
percentage of sample members (Tables 1 and 2). We find larger differences between the 
earnings levels reported in the survey and administrative data than were found in previous 
studies using similar populations. One possible explanation for this finding is that the National 
Job Corps Study was conducted during a period of strong economic growth, which may have 
increased the earnings sample members received from informal jobs. 

Annual employment rates are similar using the survey and administrative data, but 
quarterly employment rates are much higher using the survey data. The annual 
employment rate in 1998 is about 80 percent according to both the survey and SER data (Table 
1). However, the quarterly employment rates in quarters 15 and 16 after random assignment are 
substantially higher using the survey than UI data (Table 2). 

Differences in the 1998 earnings gains using the survey and SER data are due in 
roughly equal parts to reporting differences between the two data sources and to 
nonresponse bias. The estimated 1998 earnings gain is 10.4 percent according to the survey 
data and 3.9 percent according to the SER data (Table 1). Using the sample of respondents to the 
48-month interview only, the SER-based earnings gain increases from 3.9 to 6.9 percent (Table 
3), which is still smaller than the 10.4 percent survey-based figure. Thus, the residual is due to 
reporting differences between the two data sources that are slightly greater for the program than 
control group. We estimate that about 46 percent of the difference between the 1998 earnings 
gains using the survey and SER data is due to interview nonresponse bias, and 54 percent is due 
to reporting differences between the two data sources. 

EXAMINING REPORTING DIFFERENCES BETWEEN THE SURVEY AND 
ADMINISTRATIVE DATA 

We have seen that the pattern of impact findings using the administrative and survey data is 
similar in periods covered by both data sources. However, the estimated impacts are larger using 
the survey data. This is due primarily to reported earnings levels that are substantially higher 
according to the survey than administrative data for most sample members. 



xviii 




IMPACTS ON CALENDAR YEAR EARNINGS AND EMPLOYMENT RATES 
EOR THE EULL SAMPLE, BY DATA SOURCE 
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XIX 



1998 81.4 78.9 2 . 4 *** 84.5 83.2 1 . 3 ** 

1999 84.4 83.0 1 . 4 ** 78.3 77.3 1.0 

2000 83.5 82.8 0.6 75.0 77.2 - 2 . 1 ** 

2001 800 707 03 706 823 - 2 . 7 ** 

Sample Size 6,828 4,485 11,313 9,264 5,874 15,138 4,613 2,855 7,468 




Sources: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; (2) annual social security earnings 

records for the full research sample; and (3) quarterly UI earnings records from the following 22 randomly selected states for those who signed the 
records release consent form: AR, AZ, CA, FL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, TX, VA, and WA. 
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TABLE 2 



IMPACTS ON EARNINGS AND EMPLOYMENT USING SURVEY AND UI DATA 
EOR THE FULL SAMPLE, BY QUARTER AFTER RANDOM ASSIGNMENT 



Data Source 










Quarterly UI Earnings Records 






Survey Data 






from 22 States 






Program 


Control 


Estimated 


Program 


Control 


Estimated 


Outcome Measure 


Group 


Group 


ImpacL 


Group 


Group 


ImpacU 


Average Earnings, 
by Quarter After 
Random Assignment 
(in 1995 Dollars) 














1 


565.4 


851.4 


-286.0*** 








4 


1,201.0 


1,378.4 


-177 4*** 








8 


1,992.8 


1,909.8 


83.0* 








12 


2,550.7 


2,321.5 


229.2*** 








13 


2,669.4 


2,444.1 


225.4*** 








14 


2,727.6 


2,524.2 


203.5*** 








15 


2,778.3 


2,564.2 


214.1*** 


1,396.5 


1,299.1 


97.3 


16 


2,827.0 


2,591.6 


235.4*** 


1,414.8 


1,382.0 


32.8 


17 








1,449.7 


1,470.9 


-21.2 


18 








1,508.9 


1,511.6 


-2.7 


19 








1,545.6 


1,553.5 


-7.9 


20 








1,568.6 


1,593.0 


-24.4 


21 








1,632.6 


1,677.3 


-44.8 


22 








1,707.8 


1,772.0 


-64.3 


23 








1,721.7 


1,775.5 


-53.8 


24 








1,800.4 


1,857.7 


-57.3 


25 








1,856.2 


1,909.0 


-52.8 


26 








1,909.0 


1,955.6 


-46.6 


Percentage Employed, 
by Quarter After 
Random Assignment 














1 


33.2 


42.1 










4 


49.8 


57.7 


_Y 








8 


59.0 


57.9 


1.2 








12 


66.2 


63.0 










13 


66.8 


63.4 


3 4*** 








14 


67.5 


65.1 


2 4*** 








15 


69.2 


65.6 


3.6*** 


55.0 


52.6 


2.3 


16 


71.1 


68.7 


2 4*** 


55.4 


55.1 


0.3 


17 








55.2 


54.9 


0.3 


18 








54.9 


55.9 


-1.0 


19 








55.2 


56.3 


-1.1 


20 








53.6 


55.5 


-1.8 


21 








53.9 


55.5 


-1.6 


22 








55.3 


57.1 


-1.7 


23 








55.5 


57.4 


-2.0 


24 








55.3 


58.5 


-3.3** 


25 








56.5 


58.6 


-2.1 


26 








58.1 


61.0 


-2.8 


Sample Size 


6,828 


4,485 


11,313 


4,613 


2,855 


7,468 
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TABLE 2 {continued) 



Sources: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; and 

(2) quarterly UI earnings records from the following 22 randomly selected states for those who signed the records 
release consent form: AR, AZ, CA, FL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, TX, 
VA, and WA. 

Notes: I. Blank entries signify that figures are not applicable because data were not available or sample sizes were too small 

to generate precise estimates. 

2. All estimates were calculated using sample weights to account for (1) the sample design (for both data sources), 

(2) the survey design and interview nonresponse (for the survey data) and (3) the selection of states to the UI 
sample and nonresponse to the records release form (for the UI data). Standard errors of the estimates account for 
design effects due to the unequal weighting of the data and clustering of areas for in-person interviews at baseline 
(for the survey data) and the selection of states (for the UI data). 

■“These estimated impacts pertain to eligible applicants, and are measured as the difference between the weighted means for 
program and control group members. 

*Significantly different from zero at the .10 level, two-tailed test. 

**Significantly different from zero at the .05 level, two-tailed test. 

***Significantly different from zero at the .01 level, two-tailed test. 
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IMPACTS ON CALENDAR YEAR EARNINGS AND EMPLOYMENT RATES EOR RESPONDENTS 
TO THE 48-MONTH INTERVIEW, ACCORDING TO THE SURVEY AND SER DATA 
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There are several possible explanations for the higher reported earnings levels in the survey 
data. First, informal and some formal jobs are not covered by the administrative records data but 
may be captured in the survey data. Second, some survey respondents may have over-reported 
their earnings and employment levels due to recall error or other reasons. Third, some 
employers may have inaccurately reported (or not reported) sample members’ earnings to the 
government. Finally, the administrative records data may have missed earnings from sample 
members with SSNs (or other identifying information) that were incorrectly reported by 
employers or sample members. 

To examine these explanations, we compared individual employment and earnings measures 
based on the UI and survey data in quarter 16 after random assignment (the most recent 
overlapping period). We did not use the SER for these analyses, because SSA does not release 
earnings records for individuals, but only for groups of individuals. 

The analysis focused on the following questions: (1) Why are quarter 16 employment levels 
13 percentage points higher and the number of jobs per worker 20 percent higher in the survey 
data than in the UI data? and (2) Why are quarter 16 earnings levels nearly 40 percent higher in 
the survey data than in the UI data, even for those with the same number of reported jobs 
according to both data sources? 

Errors in sample members’ Social Security Numbers (SSNs) partly account for the 
higher employment levels in the survey than the UI data. Unlike SSA, UI agencies do not 
verify reported SSNs before matching to their earnings records. This problem is exacerbated by 
the fact that about 12 percent of our sample members reported multiple SSNs over the course of 
the study. Thus, the UI wage records could miss earnings from persons with SSNs that were 
incorrectly reported by employers or sample members. Our finding that employment rates and 
mean earnings are somewhat lower in the UI than the SER data support this explanation. 

The non-coverage of some formal jobs under the UI program appears to account for 
only a small portion of the gap between the employment rate as measured by the survey 
and UI data. The UI data do not cover workers in all formal jobs (for example, federal workers, 
military staff, self-employed persons, some agricultural labor, and domestic service workers). 
Using workers in the survey data, we find that those who were likely to be in non-covered formal 
jobs are somewhat less likely to have a record in the UI data than those who were likely to be in 
covered formal jobs. However, many of those likely to be in covered jobs do not have a record 
in the UI data. Thus, differences in survey and UI match rates across occupations are smaller 
than expected. These findings could be due in part to errors in classifying jobs reported in the 
survey into occupational categories, a result of limited survey information on the nature and title 
of jobs held by sample members. 

The survey data provides only weak evidence that the higher employment rate is due to 
informal jobs. Sample members with informal (casual or cash-only) jobs were asked to report 
them in the survey, but these jobs were not likely to have been reported in the UI data. To 
examine the extent to which informal jobs explain the higher survey-based employment rate, we 
compared the characteristics of jobs reported in both the survey and UI data with the 
characteristics of jobs reported in the survey data only. As expected, average hourly wages and 
the likelihood of having available fringe benefits on the job were slightly lower for the survey- 
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only group. However, job tenure and usual hours worked were similar for the two groups of 
workers. Consequently, the differences in the characteristics of jobs held by the two groups of 
workers were smaller than expected. 

Substantial unobserved factors account for the employment rate differences according 
to the survey and UI data. Few explanatory variables have predictive power in a multivariate 
regression model of whether survey-based jobs are reported in the UI data. Age, fertility status, 
marital status, health status, education level, welfare receipt status, and crime and drug use 
experiences do not significantly affect whether survey-reported jobs are reported in the UI data. 
Furthermore, only a few of the employment-related variables are statistically significant. 

The likely over-reporting of hours worked in the survey data plays an important role 
in explaining the higher earnings per job levels in the survey than UI data. The level of 
earnings over a given period is the product of (1) the number of weeks worked on the job during 
the period, (2) the usual hours per week worked, and (3) the hourly wage rate. An examination of 
the association between each of these earnings components (as measured by the survey) and the 
ratio of average survey-to-UI earnings found that the survey-to-UI ratios increase with the 
number of hours worked as reported in the survey, but not with hourly wage rates or weeks 
worked. Moreover, the average worker reported working about 42 hours per week on their most 
recent job in quarter 16, and more than three-quarters reported working at least 40 hours — 
figures that are higher than the corresponding figures for all U.S. workers. 

Some evidence suggests that earnings differences between the survey and UI data are 
smaller for those in stable jobs than less stable ones. We found some support for the 
hypothesis that earnings differences using the survey and UI data are smaller for sample 
members who held stable jobs. Earnings differences are much smaller for those with longer job 
tenure. Furthermore, the differences are somewhat larger for those in occupations that are more 
likely to have irregular hours (such as construction and private household occupations). 

Few differences in findings occur between the program and control groups. Reporting 
differences between the survey and UI data are slightly larger for the program than control 
group, resulting in percentage earnings gains that are slightly larger according to the survey than 
UI data. However, no evidence was found that the program group was more likely than the 
control group to hold informal jobs or formal jobs not covered by the UI program; the 
distribution of the occupations of the jobs held by program and control group members in quarter 
16 is very similar. Furthermore, there is no evidence that the program group was more likely 
than the control group to over-report hours worked on their jobs. 

INTERVIEW NONRESPONSE BIAS 

As discussed, we found using the SER data that post-program earnings impacts for 
48-month interview respondents are larger than for interview nonrespondents. These results 
suggest that the survey-based earnings impact estimates are biased upwards. What accounts for 
the interview nonresponse bias? The two possible explanations are: 
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1. Differences in the baseline characteristics of respondents in the program and 
control groups that are correlated with earnings. If interview respondents in the 
program group were drawn from a somewhat more advantaged subpopulation of the 
full program group than was the case for interview respondents in the control group, 
the survey-based impact estimates would be biased upwards. 

2. True differences in the earnings impacts for survey respondents and survey 
nonrespondents. If earnings impacts are truly larger for survey respondents than 
survey nonrespondents, the survey-based earnings impacts would be biased upwards 
even if the observable and unobservable characteristics of respondents in the program 
and control groups are similar. 

While it is difficult to disentangle these two possible explanations, the data support 
more strongly the explanation that the bias is caused by true differences in the earnings 
impacts for survey respondents and nonrespondents. Several pieces of evidence indicate that 
respondents in the program and control groups are comparable, suggesting that the former 
explanation cannot fully account for the nonresponse bias. First, the 48-month interview 
response rates are similar for the program and control groups. Second, the distributions of a 
large number of observable baseline characteristics and of the number of months until the 
48-month interview was completed are similar for respondents in the program and control 
groups. Third, impact estimates based on the survey data are similar for subsamples of interview 
respondents that were formed to equalize the interview response rate for the program and control 
groups by selecting those who completed interviews //rj't. Finally, impact estimates based on the 
SER data are similar using 12-month and 48-month interview respondents, even though the 
response rate was much higher to the 12-month interview. 

The available evidence suggests also that earnings impacts truly differ for interview 
respondents and nonrespondents, supporting the second explanation for nonresponse bias. 
Observable baseline characteristics differ somewhat for respondents and nonrespondents, and 
mean earnings levels using the SER data were larger for respondents than nonrespondents during 
the post-program period. Most importantly, respondents had somewhat higher Job Corps 
participation levels than nonrespondents and stayed in the program for nearly one month longer 
on average than nonrespondents. 

IMPACT FINDINGS AFTER THE PERIOD COVERED BY THE SURVEY 

Based on the administrative data, we find no impacts of Job Corps for the full sample 
on employment or earnings after the four- year period covered by the survey. The estimated 
impacts on calendar year earnings in 1999 to 2001 are all near zero and none are statistically 
significant (Tables 1 and 2). The earnings impacts in the post-survey period for 48-month 
interview respondents only are also not statistically significant (Table 3). 

However, the SER-based earnings impacts for 20- to 24-year-olds at program 
application appear to have persisted. We find no beneficial SER-based earnings impacts in 
2000 and 2001 that are statistically significant for any subgroup. However, positive earnings 
gains for those 20 to 24 and those with a high school credential at program application persisted 
with little decay. 
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BENEFIT-COST ANALYSIS 



As Job Corps is an intensive program that aims to make long-term impacts on the lives of 
the youth it serves, it is important to consider the benefits that may occur after the four-year 
survey observation period. In our initial benefit-cost analysis based only on survey data, we 
found that benefits exceed costs by $17,000 per participant (Table 4). A key assumption 
underlying this finding was that the impacts on earnings in the observation period would persist 
without decay for the rest of the average participant’s working lifetime. The impact findings 
using the administrative data, however, place the validity of this assumption in question. 

TABLE 4 

INITIAL AND REVISED ESTIMATES OE BENEEITS AND COSTS OE JOB CORPS 

(1995 Dollars) 





Initial Estimates: 

Used Survey Data and 
Earnings Impacts Assumed 
Not to Decay 


Revised Estimates: 

Used Adjusted Survey Data and Earnings 
Impacts'* Assumed to Decay at Rate Observed 
in SER Data’’ 




Eull Sample 


Eull Sample 


20-24 Year Olds 


Total Benefits 


30,957 


3,695 


14,696 


Increased Output 


27,531 


269 


17,547 


Years 1-4 


753 


-60 


588 


After Year 4 


26,778 


329 


16,959 


Other Benefits 


3,426 


3,426 


-2,850 


Program Costs 


-14,128 


-13,844 


-15,193 


Net Benefits 


16,829 


-10,150 


-496 



Source: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month 

interviews; (2) annual social security earnings records; and (3) McConnell and Glazerman (2001). 

“Earnings reported on the surveys are adjusted for survey nonresponse and overreporting of hours by 10 percent. 
The length of time youth are in Job Corps is also adjusted for nonresponse; this affects estimates of program costs 
and the output produced during vocational training in Job Corps. 

’’The rate of decay in the SER earnings impacts from the fourth year after random assignment to the seventh year 
after random assignment is 68.3 percent for the full sample and 5.9 percent for the 20 to 24 year olds. 



The revised benefit-cost estimates suggest that the benefits to society of Job Corps are 
smaller than the substantial program costs. The revised estimates are based on the estimated 
survey earnings impacts that are adjusted downward to account for nonresponse bias and the 
likely overreporting of hours. We assume also that the earnings impacts decay at the same rate 
after the observation period as the impacts based on the SER data — 68.3 percent per year. Under 
these assumptions, costs exceed benefits by $10,200 per participant (Table 4). This change in 
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findings is due to the replacement of the assumption that earnings impacts persist with an 
assumption, more consistent with the administrative data, that they will decay rapidly. The 
finding that costs exceeds benefits for the full sample holds under a wide range of reasonable 
assumptions. 

Job Corps may be cost-effective for the older youth whose earnings impacts persisted 
during the post-survey period. We find that benefits to society are only $500 lower than 
program costs for the youth who were 20- to 24-years old at program application, under the 
assumption that the positive earnings impacts in 1998 to 2001 will decay at the same rate as they 
do in the SER data for this subgroup (Table 4). While this is our best estimate of the benefits 
and costs, the finding that costs exceed benefits is sensitive to small changes in assumptions. For 
example, if we treat the positive impact on arrests for murder for this subgroup as an anomaly, 
benefits would exceed costs. 

Job Corps is still worthwhile for its participants. Job Corps is a good deal for program 
participants because the value of pay, food, and clothing they receive in the program offsets the 
earnings forgone while they are enrolled in Job Corps. Thus, the program has important 
distributional effects. 

CONCLUSIONS 

Intensive, costly programs like Job Corps can only be expected to show benefits that exceed 
costs over a relatively long time horizon. Unfortunately, the foundation of empirical evidence to 
make long-term extrapolations of the profile of earnings in response to programs such as Job 
Corps simply does not exist. Nonetheless, we are forced to make extrapolation assumptions 
about future earnings. When we conducted the initial benefit-cost analysis, we assumed that the 
earnings impact in the last year of the observation period would persist with little decay, and 
found that program benefits to society exceed program costs. The analysis of the administrative 
data, however, casts doubt on the validity of this assumption. If true earnings impacts decay at 
the same rate as observed in the administrative data, our initial conclusion is reversed — the costs 
of Job Corps exceed its benefits for the full sample, although the program may be cost-effective 
for the older youth. Job Corps is too costly a program for short-term benefits to exceed costs. 
However, we still have observed earnings for only five years after the program group left Job 
Corps. Only further long-term follow up would eliminate all uncertainties about the 
effectiveness of the program. 

The impact findings for the older youth can help guide future program improvement. Job 
Corps appears to have a longer-term beneficial effect on the earnings of older students than 
younger ones. Older students remain in Job Corps longer than younger ones, receive more hours 
of vocational training while enrolled, and are more highly motivated and well-behaved (as 
reported by program staff). Furthermore, many of the youngest sample members in the control 
group returned to high school after being rejected from Job Corps, whereas fewer older control 
group members enrolled in alternative education and training programs. These findings suggest 
that to improve overall program effectiveness. Job Corps needs to fully address differences by 
age in program structure and experience, and perhaps, to reassess the target population served by 
the program. 
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Finally, it is important to emphasize that the findings presented in this report pertain to the 
Job Corps program as it operated in 1995 and 1996 (when our program group members were 
enrolled in Job Corps), and not necessarily to the program as it operates today. There have been 
a number of significant changes that Job Corps has recently implemented in response to WIA 
provisions and other factors. For example, more Job Corps centers are now accredited to award 
high school diplomas, and Job Corps is more focused on providing longer-term support and 
placement services for their former students. These changes may have improved program 
effectiveness. 
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I. INTRODUCTION 



Job Corps stands out as the nation’s largest, most comprehensive education and job training 
program for disadvantaged youths. It serves disadvantaged youths between ages 16 and 24, 
primarily in a residential setting. The program’s goal is to help youths become more responsible, 
employable, and productive citizens. Each year, it serves more than 60,000 new participants at a 
cost of about $1.5 billion, which is more than half of all funds spent by the U.S. Department of 
Labor (DOE) on youth training programs. Because Job Corps is one of the most expensive 
education and training programs currently available to youths, DOE sponsored the National Job 
Corps Study, which has been conducted since 1993, to examine the effectiveness of the 
program.^ 

The cornerstone of the study was the random assignment of all youths found eligible for Job 
Corps to either a program group or a control group. Program group members could enroll in Job 
Corps; control group members could not, but they could enroll in other training or education 
programs. The research sample for the study consists of about 9,400 program group members 
and 6,000 control group members. These youths were randomly selected from among the nearly 
81,000 first-time applicants nationwide who applied to Job Corps from November 17, 1994, 
through December 16, 1995, and who were found eligible for the program by Eebruary 29, 1996. 

The impact findings presented in the main report for the study (Schochet et al. 2001) are 
based on the comparisons of the average outcomes of program and control group members using 
survey data collected during the four years after random assignment. These data indicate that Job 

* The evaluation was conducted by Mathematica Policy Research, Inc. (MPR) and its 
subcontractors, Battelle Human Affairs Research Centers and Decision Information Resources, 
Inc. 
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Corps generated positive impacts on earnings — the key outcome for the study — beginning in the 
third year after random assignment, and the impacts persisted through the end of the four-year 
follow-up period. Earnings gains were found for the full sample, as well as broadly across 
subgroups defined by youth characteristics at baseline and the characteristics of centers to which 
the youths were assigned. As discussed in our benefit-cost report (McConnell and Glazerman 
2001), program benefits exceed program costs under the assumption that the earnings gains 
observed during the observation period will persist with little decay. 

Administrative earnings records provide an alternative data source for obtaining information 
on the employment and earnings of program and control group members. These data allow us to 
estimate longer-term earnings impacts than could be obtained using the survey data, as well as to 
test our assumption used in the benefit-cost analysis about the persistence of future earnings 
impacts. Two sources of administrative data were collected for the evaluation: 

1. Annual Social Security Earnings Data Reported by Employers to the Internal 
Revenue Service (IRS). These Summary Earnings Record (SER) data were used to 
estimate annual earnings impacts between 1993 and 2001. 

2. Quarterly Wage Records Reported by Employers to State Unemployment 
Insurance (UI) Agencies in 22 Randomly Selected States. These data were used to 
estimate quarterly earnings impacts from 1999 to 2001. 

The administrative earnings records data were used in the Job Corps evaluation for two main 
reasons. Eirst, they were used to assess whether administrative and survey earnings data yield 
similar estimates of earnings impacts during periods covered by both data sources. The two sets 
of impact estimates could differ because of survey nonresponse bias or reporting differences in 
the two types of data. Second, they were used to obtain estimated impacts for an additional two 
and a half years beyond the four-year follow-up period covered by the survey. This postsurvey 
period is too short to make definitive conclusions about the long-term effects of Job Corps on 
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participants’ earnings. However, the data can be used for initial assessment of the 
appropriateness of the key assumption made in the benefit-cost analysis that the economic return 
to Job Corps participation will persist without decline for the rest of a youth’s working lifetime 
(McConnell and Glazerman 2001). 

This report presents impact findings using the administrative earnings data. We find that the 
pattern of earnings impacts using the administrative and survey data are similar in periods 
covered by both data sources, although the survey-based impact estimates are larger and more 
often statistically significant. This occurs primarily because reported earnings levels are 
substantially higher according to the survey data. Based on the administrative records data, we 
find no statistically significant beneficial earnings impacts after the four-year period covered by 
the survey. There is some evidence, however, of persistent earnings gains for those who were 20 
to 24 years old at program application. Finally, program costs exceed program benefits to society 
under revised assumptions about the decay of future earnings impacts based on the impact 
findings using the administrative records data. However, the program appears to be cost- 
effective for the 20- to 24-year-olds, under the assumption that the positive earnings impacts in 
1998 to 2001 persist with a slight decline for this group. Job Corps is also beneficial from the 
perspective of program participants. 

This report contains six chapters. The rest of this chapter provides an overview of Job Corps 
and the sample and survey designs for the National Job Corps Study. Chapter II discusses 
features of the UI and SER data, including the jobs they cover, our design for collecting these 
data, and the analytic methods used to estimate administrative-based earnings impacts. 
Chapter III presents the impact estimates using the administrative earnings data and compares 
them to the impact estimates using the survey data. Chapter IV presents results from analyses 
that we conducted to help understand reporting differences between the survey and 
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administrative earnings data. Chapter V presents estimates of program benefits and costs that 
incorporate the impact findings using the administrative data. Chapter VI contains concluding 
remarks. 

A. OVERVIEW OF JOB CORPS 

The Job Corps program, established by the Economic Opportunity Act of 1964, operated 
under provisions of the Job Training Partnership Act (JTPA) of 1982 during the study period.^ 
Job Corps uses a well-defined program model (documented in Johnson et al. 1999), which had 
been refined continually over 30 years at the time our study sample attended in 1995 and 1996, 
and which has continued to evolve since the study period. Because many Job Corps centers are 
some distance away from the home areas of the students who attend the centers, different 
organizations have traditionally performed three key programmatic functions. These functions 
are (1) recruiting and screening students, (2) operating center programs, and (3) helping youths 
find jobs or further training after they leave Job Corps. 

A complex operational structure supports the program. This structure has many levels of 
administrative accountability and numerous contractors and subcontractors. DOE administers 
Job Corps through a national office and nine regional offices. The national office establishes 
policy and requirements, develops curricula, and oversees major program initiatives. One 
example of a national office initiative is the continual development of the Job Corps performance 
measurement system, which has been in place for nearly two decades. 

Regional offices of DOE procure and administer contracts and perform oversight activities, 
such as reviews of center performance. DOE uses a competitive bidding process to contract out 

^ Beginning in July 2000, Job Corps has operated under provisions of the Workforce 
Investment Act of 1998. 
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center operations, recruitment and screening of new students, and placement of students into jobs 
and other educational opportunities after they leave the program. At the time of the study, 80 
centers were operated under competitive contracts. In addition, the U.S. Department of 
Agriculture and the U.S. Department of the Interior operated 30 Civilian Conservation Centers 
(CCCs) under interagency agreements with DOL.^ Job Corps centers are in all regions of the 
country and in most states. There were 105 Job Corps centers that were operating in the 48 
contiguous states when program group members were enrolled in Job Corps.”* 

Next, we briefly describe the three main program elements: (1) outreach and admissions, 
(2) center operations, and (3) placement. 

1. Outreach and Admissions 

Outreach and admissions (OA) agencies provide information to the public through outreach 
activities, describe the program to youths who apply, and screen youths to ensure that they meet 
the eligibility criteria. They also assign youths to centers (when the regional office delegates this 
function) and arrange for transportation to centers. OA agencies include private nonprofit firms, 
private for-profit firms, state employment agencies, and the centers themselves. At the time of 
the study, 41 percent of all students were screened by private organizations that were not centers, 
30 percent were screened by centers that also held an OA contract, and 29 percent were screened 
by state employment security agency personnel. The use of these various types of OA agencies 
varied widely across regions (see Johnson et al. 1999). 



** Currently, 90 contract centers and 28 CCCs are providing Job Corps training. 
”* Five centers in Alaska, Hawaii, and Puerto Rico were not part of the study. 
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2. Center Operations 

Centers are the heart of the Job Corps program. Each center provides comprehensive, 
intensive services that include basic education, vocational training, residential living, health care 
and education, and counseling. 

Education. Education programs in Job Corps are individualized and self-paced, and they 
operate on an open-entry and open-exit basis. The programs include remedial education 
(emphasizing reading and mathematics), world of work (including consumer education), driver’s 
education, home and family living, health education, programs designed for those whose primary 
language is not English, and a General Educational Development (GED) program of high school 
equivalency for academically qualified students. About one-fifth of the centers can grant state- 
recognized high school diplomas. 

Vocational Training. As with the education component, the vocational training programs at 
Job Corps are individualized and self-paced and operate on an open-entry, open-exit basis. Each 
Job Corps center offers training in several vocations, typically including business and clerical, 
health, construction, culinary arts, and building and apartment maintenance. National labor and 
business organizations provide vocational training at many centers through contracts with the Job 
Corps national office. Union members teach these classes at the centers. 

Residential Living. The residential living component distinguishes Job Corps from all 
other publicly funded employment and training programs. The idea behind residential living is 
that, because most participants come from disadvantaged environments, they require new, more 
supportive surroundings to derive the maximum benefits from education and vocational training. 
All students must participate in formal social skills training. The residential living component 
also includes meals, dormitory life, entertainment, sports and recreation, center government, 
center maintenance, and other related activities. Historically, regulations had limited the number 
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of nonresidential students to 10 percent, but Congress raised that limit to 20 percent in 1993. 
About 12 percent of Job Corps study program group participants were nonresidential students. 

Health Care and Education. Job Corps centers offer comprehensive health services to 
both residential and nonresidential students. Services include medical examinations and 
treatment; biochemical tests for drug use, sexually transmitted diseases, and pregnancy; 
immunizations; dental examinations and treatment; counseling for emotional and other mental 
health problems; and instruction in basic hygiene, preventive medicine, and self-care. 

Counseling and Other Ancillary Services. Job Corps centers provide counselors and 
residential advisers. These staff members help students plan their educational and vocational 
curricula, offer motivation, and create a supportive environment. Support services are also 
provided during recruitment, placement, and the transition to regular life and jobs following 
participation in Job Corps. 

3. Placement 

The final step in the Job Corps program is placement, which helps students find jobs in 
training-related occupations with prospects for long-term employment and advancement. 
Placement contractors may be state employment offices or private contractors; sometimes, the 
centers themselves perform placement activities. Placement agencies help students find jobs by 
providing assistance with interviewing and resume writing and services for job development and 
referral. They also distribute the readjustment allowance, a stipend students receive after leaving 
Job Corps. 

B. OVERVIEW OF THE SAMPLE AND SURVEY DESIGNS FOR THE NATIONAL 

JOB CORPS STUDY 

In this section, we highlight key features of the sample and survey designs for the Job Corps 
evaluation, including the labor market information collected during the interviews (for a detailed 
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discussion, see Schochet et al. 2001 and Schochet 2001). This information is needed to fully 
understand the design for the administrative earnings records study discussed in Chapter II. 

1. Sample Design 

The central feature of the study design was the random assignment of all youths found 
eligible for Job Corps, either to a program group whose members were permitted to enroll in Job 
Corps or to a control group whose members were not. Sample intake occurred between 
November 1994 and February 1996. With few exceptions, all youths who applied to Job Corps 
for the first time between November 16, 1994, and December 17, 1995, and were found eligible 
for the program were included in the study — a total of 80,883 eligible applicants. During the 
sample intake period, 5,977 Job Corps-eligible applicants were randomly selected to the control 
group. Approximately 1 eligible applicant in 14 (7 percent of 80,883 eligible applicants) was 
assigned to the control group. 

During the same 16-month period, 9,409 eligible applicants were randomly assigned to the 
research sample as members of the program research group (hereafter called the program 
group). ^ Because random assignment occurred after youths were determined eligible for Job 
Corps (and not after they enrolled in Job Corps centers), the program group included youths who 
enrolled in Job Corps (about 73 percent of eligible applicants), as well as those who did not 
enroll, the so-called “no-shows” (about 27 percent of eligible applicants). Although the study 
focused on enrollees, all youths who were randomly assigned, including those who did not enroll 
at a center, were included in the analysis to preserve the benefits of the random assignment 
design. 

^ The remaining 65,497 eligible applicants were randomly assigned to a program 
nonresearch group. These youths were allowed to enroll in Job Corps but were not in the 
research sample. 
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Control group members were not permitted to enroll in Job Corps for three years, although 
they were able to enroll in other programs available to them. Thus, the outcomes of the control 
group represent the outcomes that the program group would have experienced if they had not 
been given the opportunity to enroll in Job Corps. Because control group members were allowed 
to enroll in other education and training programs, the comparisons of program and control 
group outcomes represent the effects of Job Corps relative to other available programs that the 
study population would enroll in if Job Corps were not an option. The impact estimates do not 
represent the effect of the program relative to no education or training; instead, they represent the 
incremental effect of Job Corps. 

The National Job Corps Study is based on a fully national sample. With a few exceptions, 
the members of the program and control groups were sampled from all OA agencies located in 
the contiguous 48 states and the District of Columbia, rather than from only some OA agencies 
in certain areas.'’ This design feature allowed us to obtain impact estimates that are more precise 
than those that could be obtained from a clustered sample of the same size. In addition, the 
nonclustered design spread the burden of random assignment across all OA agencies and Job 
Corps centers, which reduced the burden on any one agency or center. 

The sampling rates to the control and program groups differed for some population 
subgroups for both programmatic and research reasons. For example, OA agencies had 
difficulty recruiting females for residential slots, and Job Corps staff were concerned that the 
presence of the control group would cause these slots to go unfilled. Therefore, sampling rates to 
the control group were set lower for females in areas from which high concentrations of 

Youths who previously participated in Job Corps (“readmits”) or applied for one of seven 
small, special Job Corps programs were excluded from the study (see Burghardt et al. 1999 for 
further discussion of these groups). 
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residential students come. Because of differences in sampling rates across population subgroups, 
all analyses were conducted using sample weights so that the impact estimates can be 
generalized to the intended study population: applicants in the 48 contiguous states and the 
District of Columbia who applied to Job Corps during the 13-month period between 
November 17, 1994, and December 16, 1995, and who were determined to be eligible for the 
program.’ 

As expected, random assignment produced program and control groups whose distributions 
of characteristics prior to random assignment were similar (Schochet 1998). In addition. Job 
Corps staff implemented random assignment procedures well (Burghardt et al. 1999). Weekly 
extracts from the Job Corps management information system on all new center enrollees showed 
that less than 0.6 percent of enrollees arrived at a center without having been previously 
randomly assigned; thus, nearly all those in the study population were subject to random 
assignment. Furthermore, only 1.4 percent of control group members enrolled in Job Corps 

o 

before the end of the three-year period during which they were not supposed to enroll. 

2. Survey Design 

Interviews with the research sample were conducted at baseline (shortly after random 
assignment) and at 12, 30, and 48 months after random assignment. Interviews were conducted 
by telephone and in person for those not reachable by telephone. During the baseline interview, 
to conserve data collection costs, in-person interviews were conducted in randomly selected 

’ The study population also included only those whose random assignment forms were 
received by MPR before March 1, 1996. This restriction did not exclude many eligible 
applicants who applied to the program during the 13-month period, because the time between 
program application and eligibility determination is typically very short. 

o 

An additional 3.2 percent of control group members enrolled in Job Corps after their three- 
year restriction period ended and before four years after random assignment. 
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areas only, which resulted in a slightly clustered survey sample (Schochet 2001). Youths in areas 
not selected for in-person interviewing at baseline and who did not complete baseline interviews 
were not eligible for follow-up interviews (because they were fully represented by those in the 
in-person areas who completed in-person interviews). For the follow-up interviews, however, in- 
person interviews were conducted with eligible sample members in all areas. 

The sample used in the main impact report includes 11,313 youths (6,828 program group 
and 4,485 control group members) who completed a 48-month interview. The effective response 
rate to the 48-month interview (that is, the response rate in areas selected for in-person 
interviewing at baseline) was 79.9 percent (81.5 percent for the program group and 77.8 percent 
for the control group). ^ The distributions of baseline characteristics of program and control group 
members in the 48-month sample are similar (Schochet 2001).^'^ 

3. Labor Market Information Collected in the Surveys 

Each interview requested detailed information on the labor market experiences of 
respondents since the previous interview. We collected labor market information for up to 2 
jobs that were in progress at the last interview date, and up to 10 new jobs that started since the 
last interview date. We asked respondents to provide information on paid full-time or part-time 



^ Of the 9,937 youths in the in-person areas at baseline who were eligible for 48-month 
follow-up interviews, 7,940 completed 48-month interviews. 

The distribution of baseline characteristics, however, differ slightly for those in the 48- 
month sample and the nonrespondents. Thus, we adjusted the sample weights using propensity 
scoring procedures so that the observable baseline characteristics of the 48-month sample 
matched those of the full sample (Schochet 2001). 

' ' The baseline interview collected labor market information since the year prior to random 
assignment. 
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jobs that they had, including odd jobs, paid baby-sitting jobs, military service, work in their own 
businesses, and other types of jobs they may have had on a regular basis. 

The surveys requested the following key labor market information for each job respondents 
held since the previous interview: 

• Start and end dates 

• Periods of two weeks or longer when they were not working and not getting paid 

• Days per week usually worked 

• Hours per day usually worked, including overtime hours 

• Kind of company worked for (coded into three-digit industry [SIC] codes) 

• Nature of the work and job title (coded into three-digit occupation [SOC] codes) 

• Type of employer — whether (1) working for a private company; (2) on active military 
duty; (3) a federal, state, or local government employee; (4) self-employed; 

(5) working without pay or in a family business or as a favor, or (6) working for 
another type of employer 

• Most recent hourly rate of pay before taxes and deductions, including tips, 
commissions, and regular overtime pay 

• Earnings per pay period for those not paid by the hour 

• Benefits available on the job (such as health insurance, paid sick leave, paid vacation, 
and retirement or pension benefits) 

To obtain consistent data across interviews, we asked the same questions at each follow-up 
interview. 

Using data on the start and end dates of jobs, we constructed weekly employment timelines 
to determine whether a sample member was working in a given week during the 208-week (that 
is, four-year) follow-up period. We then used these employment timelines and information on 
hours worked and hourly wages received on each job to construct weekly hours and earnings 
timelines. The labor market outcome measures used in the impact analysis were constructed 
using these timelines. 
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II. DESIGN OF THE ADMINISTRATIVE EARNINGS RECORDS STUDY 



We used two data sources for the administrative earnings record study: (1) social security 
earnings data reported by employers to the IRS (SER data), and (2) wage records reported by 
employers to state UI agencies. We used these data to address the following questions: 

• Do administrative and survey earnings data yield similar impact estimates on 
employment and earnings during the periods that both data sources cover? 

• Are the administrative and survey impact estimates more similar for some key 
population subgroups (defined, for example, by gender, age, race/ethnicity, 
educational level, arrest history, and residential/nonresidential status) than for others? 

• What are estimated impacts on earnings and employment in the two and a half years 
beyond the four- year period covered by the survey? 

The impact results using the survey and administrative data could differ for several reasons. 
First, nonresponse to the baseline and follow-up surveys could affect the survey-based earnings 
impact estimates. The response rate to the 48-month follow-up interview was similar for the 
program and control groups (81.5 percent for the program group and 77.8 percent for the control 
group), and the distributions of observable baseline characteristics of survey respondents in the 
two research groups are also similar (Schochet 2001). However, survey nonresponse bias could 
arise because of potential differences in (1) true earnings impacts for survey respondents and 
survey nonrespondents, or (2) unobservable baseline characteristics between survey respondents 
in the program and control groups that are correlated with earnings. We examined the extent of 
survey nonresponse bias by comparing impact estimates for respondents and nonrespondents 
using the administrative records data. 

Second, the impact results using the survey and administrative records data could differ 
because of reporting differences in the two types of data. Administrative records do not cover all 
types of jobs. Furthermore, during the survey, some sample members may not have accurately 
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recalled their earnings over a long follow-up period, and some may have systematically over- or 
underreported them. To examine the extent of reporting differences by data source, we compared 
earnings levels as measured in the survey and administrative data in overlapping periods using 
the sample of respondents to the 48-month interview. 

Because of several important differences in the way that SER and UI data are stored and 
released, we used different data collection designs for each data source. SER data are stored 
centrally, and UI data are stored separately by state. Thus, the SER data contain sample 
members’ earnings from all states, whereas, to conserve costs, the UI data contain sample 
members’ earnings from 22 randomly selected states only. In addition, for confidentiality 
reasons, the Social Security Administration (SSA) releases summary earnings measures only for 
groups of people — it does not release earnings records for individuals. Thus, we could not 
conduct detailed analyses comparing SER to survey data at the individual level. State UI 
agencies, however, release wage records for individual sample members, so that analyses 
comparing UI to survey data could be conducted at the individual level. 

These important differences in the data collection designs for the SER and UI data 
influenced the types of analyses we could conduct and the analytic methods we could use to 
estimate earnings impacts using each data source. We discuss these issues in the rest of this 
chapter. 

A. SOCIAL SECURITY EARNINGS DATA 

Each year, employers are required to report to the IRS the total earnings of each worker for 
whom social security withholdings must be made. The data reported are total earnings subject to 
social security for the calendar year. The SSA receives these data from the IRS to determine 
eligibility for social security. The SER — the data used for the study — is an extract from the 
Master Earnings Eile (MEE) maintained by the SSA. The MEE contains earnings records for 
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each worker with a social security number (SSN) who has worked in covered employment. The 
primary source for the earnings data is the W-2 form, which is sent directly to the SSA and 
updated weekly. The SER provides Federal Insurance Contribution Act (FICA)-covered wages 

1 9 

(wages in covered employment up to the annual taxable maximum). 

Annual earnings data for a calendar year are available approximately 11 months after the 
end of the respective tax year. Thus, for example, data pertaining to calendar year 1999 became 
available in late November 2000, and data pertaining to calendar year 2000 became available in 
late November 2001. The SSA also stores and releases retrospective earnings data. 

1. Coverage and Accuracy 

About 96 percent of all workers in employment or self-employment are covered under the 
Old-Age, Survivors, and Disability Insurance (OASDI) program (Social Security Administration 
2001). Workers not covered fall into five categories: (1) civilian federal employees hired before 
1984, (2) railroad workers, (3) some employees of state and local governments who are covered 
under their employers’ retirement systems, (4) people with net annual earnings from self- 
employment below $400, and (5) domestic workers and farm workers with low earnings. 
Clearly, earnings from casual jobs or the underground economy also are not covered in the SER 
data, although they may be reported in the surveys. 

On the basis of these reporting rules, the SER data contain earnings from most formal jobs 
our sample members held. The follow-up survey data indicate that, among the most recent jobs 
held by sample members in quarters 10 and 16 after random assignment, only 5 percent reported 

The maximum is updated each year. In 2001, it was $80,400. 

In 2001, domestic workers must have earned $1,300 annually from any single employer 
before FICA taxes were withheld. The corresponding figure for agricultural workers was $2,500. 
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that they were self-employed, 7 percent reported working in private household occupations, 
6 percent reported working in state or local government, and less than 2 percent reported 
working in agricultural occupations (Schochet et al. 2001). These figures are similar for the 
program and control groups. Furthermore, most self-employed, domestic, and agricultural 
workers had enough earnings to be covered under the OASDI program; average weekly earnings 
reported in the survey were about $375 for self-employed workers, $230 for those in private 
household occupations, and $300 for those in agricultural occupations. Thus, the earnings of only 
a small number of sample members who held formal jobs should be excluded from the SER data. 

The SER data are likely to be accurate, because there are financial incentives for employers 
to report earnings to the IRS correctly. Earnings employers report to the IRS can be counted as a 
business expense that can be deducted from the firm’s earnings and, thus, will lower its income 
tax.*"^ Eurthermore, few sample members had reported annual earnings that were capped at the 
annual taxable maximum for any years between 1993 and 2001. 

2. Obtaining SER Data 

To protect confidentiality, the SSA does not release earnings data for individuals. Instead, it 
provides summary earnings statistics for groups of people. Accordingly, we requested that the 
SSA run computer programs that we provided to estimate earnings impacts (that is, differences 
in mean earnings between program and control group members) and their associated levels of 
statistical significance for the full sample and for key subgroups of the Job Corps population. 
The SSA sent us the output from these computer runs. This procedure ensured that the 
information that the SSA provided could not be associated with an individual sample member. 

There could, however, be financial incentives for a firm to underreport earnings to the IRS 
if that firm evades paying taxes. 
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because all “group” means that we requested were estimated using at least 25 sample members, 
and most were estimated using at least 1,000 sample members.'^ 

We sent the SSA a data file containing identifying information (SSN, name, date of birth, 
and sex) and other data needed to run our computer program (such as sample weights and 
subgroup indicator variables) for the, full research sample of 15,301 youths (9,365 program group 
and 5,936 control group members)/^ We obtained the identifying information from the Job 
Corps intake (ETA-652) forms and the interviews. Because the identifying information 
sometimes changed over time, the data file contained, for each youth, one record for every 
possible combination of SSN, name, and date of birth that the youth ever reported. We used 
this procedure to increase match rates to the SSA data system. 

SSA procedures require that SSNs be verified before being matched to the SER data. Thus, 
the SSA used the identifying information in our data file to check the validity of each SSN using 
its Enumeration Verification System (EVS). Because our data file sometimes contained more 
than one record per sample member (as discussed above), we asked the SSA to determine the 



We also gave the SSA an alternative data collection option, where we would provide the 
SSA with an earnings group number (1 to 71) for each sample member, and it would provide us 
with mean earnings and sum of squared earnings for each of the 71 earnings groups. The 71 
earnings groups were constructed so that we could estimate earnings impacts and their standard 
errors that account for unequal weighting of the data. SSA staff, however, preferred that we send 
them the computer programs. We also preferred this option, because it allowed us to obtain 
impact estimates for a much larger set of subgroups than could be obtained under the alternative 
option. This is because the earnings groups were formed primarily so that the impact estimates 
could be weighted correctly. Thus, we could not form earnings groups using many subgroup 
indicator variables, because some of the resulting cell sizes would have been too small. 

The initial research sample contained 15,386 youths. However, we excluded 77 readmits 
(40 program and 37 control) who were determined after random assignment to have enrolled in 
Job Corps prior to random assignment and thus were excluded from the study universe. We also 
excluded an additional 8 youths for whom we did not have an SSN. 

The data file contained 20,586 observations. About 77 percent had only one record in the 
file, 18 percent had two records, and 5 percent had three or more records. 
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best match for each person and to store only one record per person (that is, one record per unique 
ID that MPR assigned to each youth at random assignment) when matching to the SER database. 

The EVS system validated SSNs for 98.9 percent of the sample (98.9 percent for the 
program group and 99.0 percent for the control group). In total, matches were found for 15,138 
of the 15,301 youths in our sample. Because of the sophisticated validation process the SSA used 
and the high match rate, we believe that the correct SSNs for our sample were matched to the 
SER database. 

After merging the SER data to the data file containing sample weights and subgroup 
indicator variables, the SSA ran our computer programs to estimate impacts on annual earnings 
between 1993 and 2001. To ensure that random assignment had produced program and control 
groups with similar mean earnings levels during the period prior to random assignment, we 
requested figures for 1993 and 1994. The estimated impacts for 1995 to 2001 pertain to the post- 
random assignment period and are the focus of this report. SSA staff ran our computer programs 
in mid- 2002 (after the 2000 SER data were finalized) and in mid- 2003 (after the 2001 SER data 
were finalized). 

3. Outcome Measures 

We used the SER data to obtain estimated impacts on annual earnings and employment rates 
in calendar years 1993 to 2001 (where annual earnings measures were scaled into 1995 dollars 
using the GDP price deflator). Random assignment for the evaluation, however, took place 
during a 14-month period between mid-November 1994 and Eebruary 1996. Thus, outcomes 
measured in “calendar time” differ from outcomes measured in “random assignment time” (that 

I 8 

Earnings were put into 1995 dollars, because many program group members were 
enrolled in Job Corps in 1995. Thus, most program costs were incurred in 1995. Consequently, 
program benefits were scaled also into 1995 dollars in the benefit-cost analysis. 
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is, in the time since random assignment). For example, calendar year earnings in 1996 represent 
earnings accrued between 7 and 18 months after random assignment for those who were 
randomly assigned in June 1995, but they represent earnings accrued between 1 and 12 months 
after random assignment for those who were randomly assigned in December 1995. 

This timing problem somewhat complicates the interpretation of the SER-based earnings 
impact estimates, because at a particular calendar date, the time of “exposure” to the intervention 
differed across sample members. However, it is fairly accurate to consider calendar year 1996 to 
be the “first” year (“year 1”) after random assignment for most sample members (because about 
55 percent of the sample was randomly assigned after June 1995) and part of year 2 for some. 
Hence, calendar year 2001 can be considered year 6 for most sample members and part of year 7 
for others. 

This categorization is roughly consistent with our definition of the “in-program” and 
“postprogram” periods used in the main impact report. In that report, we showed that years 1 and 
2 after random assignment were a period of intensive Job Corps participation for the program 
group, and years 3 and 4 were largely a postprogram period. Similarly, many program group 
members reported in the survey that they ever enrolled in Job Corps in 1995 and 1996 (about 
72 percent in 1995 and 45 percent in 1996). Thus, 1995 and 1996 can be considered in-program 
periods. Job Corps enrollment rates were about 11 percent in 1997, 3 percent in 1998, and less 
than 2 percent in 1999. Thus, the period starting in mid- 1997 was a postprogram period for 
nearly all program group members. 

In the main impact report, the survey-based earnings impacts were estimated for earnings 
measured in random assignment time. Thus, to directly compare the survey-based and SER- 
based impact estimates, we also constructed survey-based annual earnings measures in calendar 
time. These calendar time measures, however, could be constructed only using those whose 
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follow-up period covered by the 48 -month interview included the calendar year under 
investigation. For example, we could construct total 1999 earnings using the survey data only for 
those who were (1) randomly assigned in 1996, or (2) randomly assigned before 1996 but whose 
48-month interviews were conducted in 2000. 

Survey-based calendar year measures for 1996 to 1998 could be constructed for nearly all 
sample members (see Table II. 1). However, the 1995 and 1999 measures could be constructed 
only for a small number of sample members; therefore, we do not present impact estimates for 
them. 

4. Analytic Methods 

The computer programs sent to the SSA estimated impacts on earnings and employment 
outcomes by computing differences in mean outcomes between all program and control group 
members. The random assignment design ensures that this approach yields unbiased estimates of 
the effect of Job Corps for program applicants who were determined eligible for the program. 
We calculated all estimates using sample weights to account for the sample design. The standard 
errors were inflated to account for design effects due to unequal weighting of the data. The 
impact estimates were not regression-adjusted for simplicity, because the baseline characteristics 
of the program and control groups are very similar, and because of large sample sizes that 
yielded precise impact estimates using the simple differences-in-means approach. 

The comparison of the outcomes of all program and control group members yields combined 
impact estimates for the 73 percent of program group members who enrolled in Job Corps and 
the 27 percent who did not. In our main impact report, we present impact estimates per eligible 
applicant, as well as for only those who enrolled and received services (that is, for participants 
only). The impacts per participant were obtained by dividing the impact estimates per eligible 
applicant by the proportion of program group members who enrolled in Job Corps. Because the 
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TABLE II. 1 



SAMPLE SIZES EOR MEASURES BASED ON CALENDAR YEAR AND TIME SINCE 
RANDOM ASSIGNMENT, BY DATA SOURCE 









Data Source 








Survey Data 


Annual Social Security 
Earnings Records 


Quarterly UI Earnings 
Records^ 


Time Period 


Program 

Group 


Control 

Group 


Program 

Group 


Control 

Group 


Program 

Group 


Control 

Group 


Calendar Year 


1995 


295 


185 


9,264 


5,874 


0 


0 


1996 


6,627 


4,328 


9,264 


5,874 


0 


0 


1997 


6,828 


4,485 


9,264 


5,874 


0 


0 


1998 


6,808 


4,474 


9,264 


5,874 


0 


0 


1999 


1,115 


704 


9,264 


5,874 


4,613 


2,855 


2000 


0 


0 


9,264 


5,874 


4,613 


2,855 


2001 


0 


0 


9,264 


5,874 


4,613 


2,855 


QuarterA^ear Since 
Random Assignment 

Quarter 


13 


6,828 


4,485 


0 


0 


680 


387 


14 


6,828 


4,485 


0 


0 


1,982 


1,238 


15 


6,828 


4,485 


0 


0 


3,095 


1,893 


16 


6,692 


4,358 


0 


0 


3,999 


2,480 


17 


1,738 


1,079 


0 


0 


4,613 


2,855 


18 


576 


332 


0 


0 


4,613 


2,855 


19 


186 


94 


0 


0 


4,613 


2,855 


20 


68 


28 


0 


0 


4,613 


2,855 


21 


12 


3 


0 


0 


4,613 


2,855 


22 


0 


0 


0 


0 


4,613 


2,855 


23 


0 


0 


0 


0 


4,613 


2,855 


24 


0 


0 


0 


0 


4,593 


2,843 


25 


0 


0 


0 


0 


3,933 


2,468 


26 


0 


0 


0 


0 


2,631 


1,617 


27 


0 


0 


0 


0 


1,518 


962 


28 


0 


0 


0 


0 


614 


375 


Year 


4 


6,692 


4,358 


0 


0 


680 


387 


5 


68 


28 


0 


0 


4,613 


2,855 


6 


0 


0 


0 


0 


4,593 


2,843 


7 


0 


0 


0 


0 


614 


375 



Source: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month 

interviews; (2) annual social security earnings records for the full research sample; and (3) quarterly UI 
earnings records from the following 22 randomly selected states for those who signed the records release 
consent form; AR, AZ, CA, EL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, 
TX, VA, and WA. 

“Sample pertains to those who lived in one of the 22 states selected for the UI study at application to Job Corps. 
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Job Corps enrollment rate was high, the estimated impacts for participants and eligible applicants 
are similar (Schochet et al. 2001). 

We could not estimate impacts per participant using the SER data, however, because 
information on Job Corps enrollment was obtained from the survey and hence is unavailable for 
survey nonrespondents. Thus, we present SER-based impacts per eligible applicant only. 

We used the SER data to estimate impacts for the full sample (that is, for all program and 
control group members) and for the following key population subgroups: 



• Respondents and Nonrespondents to the 48-Month Interview. We conducted this 
analysis to examine the extent to which interview nonresponse affected the survey- 
based impact estimates. 

• Subgroups Defined by Key Youth Characteristics at Baseline. These characteristics 
are (1) gender, (2) age at application to Job Corps, (3) race and ethnicity, (4) arrest 
history at random assignment, (5) educational level at random assignment, and (6) the 
presence of children (for females). 

• Residents (the 88 Percent of Students Who Live at a Job Corps Center) and 
Nonresidents (the 12 Percent of Students Who Live at Home but Attend Job Corps 
During the Day). We estimated impacts by residential status using data on the 
predictions of Job Corps OA counselors as to whether sample members would be 
assigned to a residential slot or to a nonresidential one. This information was 
collected at program application (which was before random assignment) and thus is 
available for both program and control group members. These predictions closely 
matched actual residential status (Schochet et al. 2001). 

• Subgroups Defined by the Characteristics of Centers the Youths Attended. These 
center characteristics are type of operator (CCC or contract centers), size, region, and 
performance ranking (for further description of center characteristics, see Burghardt 
and Schochet 2001). These impacts were obtained using data on the predictions of 
OA counselors as to which center each youth was likely to attend. We examined 
impacts for small centers (225 slots or fewer; serving 20 percent of students), medium 
centers (226 to 495 slots; serving 46 percent of students), and large centers (more 
than 495 slots; serving 34 percent of students). High-performing centers were 
defined as those that were in the top third of the performance ranking during program 
years 1994 and 1995. Similarly, low-performing centers are those that were in the 
bottom third of the performance ranking in both years. The high- and low-performing 
groups each included just under one-fourth of centers. The remaining centers were 
designated medium-performing centers. 



Appendix Table B.l displays subgroup sample sizes using the survey and SER data. 
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We estimated impacts for a subgroup by comparing the distribution of outcomes of program 
and control group members in that subgroup. For example, we obtained impact estimates for 
males by comparing the mean outcomes of male program and control group members. Similarly, 
we obtained impact estimates for residents by comparing the mean outcomes of program and 
control group members who were designated for residential slots. 

To examine reporting differences in the survey and SER data, our computer programs 
calculated, for each 48-month survey respondent, the difference between calendar-year earnings 
according to the survey and SER data. The program produced summary statistics on these 
differences separately for the program and control groups. 

B. UI WAGE RECORDS 

UI data provide an alternative data source for obtaining information on the employment and 
earnings of youths in the National Job Corps Study. Employers in most states are required to 
maintain and submit earnings records to the state’s UI system for workers in jobs covered by 
UI.^^ These records, which are maintained in machine-readable form, are used to determine 
individual workers’ eligibility for unemployment compensation if they are laid off. 

Wage records contain calendar-quarter information, by employer, on earnings. Because it 
takes time to collect and process the data, wage records for a given calendar quarter are, 
however, generally not complete for about six months. Thus, data pertaining to the last quarter 
of 1999, for example, became available in mid-2000. Eurthermore, because the data needed to 
determine UI eligibility pertain to a base period that is usually defined as the first four of the last 



Employers in all states submit wage records. However, in a few states, wage records are 
not used to determine UI eligibility. In these cases, the UI system does not maintain wage 
records. 
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five completed calendar quarters, wage records are typically maintained online for, at most, six 
or seven quarters. Data for earlier quarters are archived and generally less readily available. 

For analyses, wage records have some advantages over the SER data. First, UI wage 
records cover calendar quarters rather than full calendar years, so they can be used to construct 
earnings measures over periods since random assignment (that is, in random assignment time). 
Second, unlike the SER data, the UI agencies provide data for individuals, so the UI data can be 
used to conduct detailed analyses examining individual differences in Ul-based and survey-based 
earnings measures. 

The main disadvantage of the UI wage records data is that they are not stored centrally 
across states; instead, they must be collected separately from each state. Making requests to 
every state and dealing with individual state access protocols make acquisition costs very 
expensive relative to the SER data. Therefore, we selected a sample of states for the UI study. 
Hence, sample sizes for the analysis of UI data were smaller than for the analysis of SER data, 
resulting in less precise impact estimates using the UI data. For this reason, we used the UI data 
to obtain impact estimates for the full sample but not for subgroups of youths. 

Another disadvantage of the UI wage records is that states often require written consent 
before releasing individual UI wage records. Thus, as described below, we obtained UI wage 
data only for the 80 percent of sample members who signed a consent form for records release 
when they applied to Job Corps. Thus, earnings impacts estimated using the UI wage records 
may not be generalizable to the full population of eligible applicants included in the National Job 
Corps Study, because of potential differences in the unobservable characteristics of those who 
signed the consent form and those who did not. 
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1. Coverage and Accuracy 

UI wage records consist of total quarterly earnings reported by employers to state UI 
agencies for each employee. By law, most employers are subject to a state UI tax and must report 
what is paid to each employee, including regular earnings, overtime, and tips and bonuses. In 
most states, the Federal Unemployment Tax Act (FUTA) applies to employers who (1) paid 
wages of $1,500 or more during any calendar quarter in the current or preceding calendar year, 
or (2) employed at least one worker for at least one day in each of the 20 weeks during the 
current or preceding calendar year. 

Most workers are covered under FUTA, and coverage provisions are similar to those under 
the OASDI program. The major difference between coverage in the two programs is that UI 
wage records do not cover federal workers, military staff, or self-employed people (the SER data 
cover self-employed people who have annual net earnings of at least $400).^'^ Another difference 
is that the SER data do not cover some state and local employees, whereas the UI wage records 
cover all state and local employees. Other workers excluded from coverage under the EUTA 
provisions include railroad employees, workers in service for relatives, most agricultural labor 
(except workers on large farms), domestic service workers whose employers paid less than 
$1,000 in wages in any calendar quarter, part-time employees of nonprofit institutions, some 
students employed by their schools, insurance and real estate agents on commission, and casual 
labor not in the course of the employer’s business (U.S. Department of Eabor 2002). Many of 
these same workers are also excluded from the OASDI program. UI wage records cover about 
94 percent of workers (U.S. General Accounting Office 2002). Thus, we expect that the UI data 

90 

Eederal workers and military staff are eligible to receive UI benefits; however, their 
earnings are not contained in the UI wage records. 
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contain earnings information on most formal jobs held by our sample members in the states 
selected for UI data collection. 

Although coverage is similar for the UI and OASDI programs, the UI wage records are 
likely to be less accurate than the SER data, for several reasons. First, employers have financial 
incentives to underreport earnings to state UI programs, because earnings reported to UI agencies 
provide the basis for assessing a payroll tax that finances UI benefit payments. In contrast, as 
discussed earlier, employers have incentives to report earnings fully to the IRS, because reported 
earnings can be counted as a business expense that can be deducted from the firm’s earnings and 
hence will lower its income tax. Second, unlike the SSA, state UI agencies do not verify 
reported SSNs. Thus, the UI wage records could miss earnings from people with SSNs that were 
incorrectly reported by employers or sample members. This problem is exacerbated by the fact 
that about 12 percent of our sample members reported multiple SSNs over the course of the 
study. For these cases, we could send only one SSN to the state UI agencies to match to their 
data systems. Consequently, the state UI wage records could have also missed earnings for cases 
for whom we selected the incorrect SSN. 

2. Obtaining UI Wage Records 

The research sample for the National Job Corps Study is based on a fully national sample of 
eligible Job Corps applicants; the program and control groups contain youths who lived in each 
of the contiguous 48 states and the District of Columbia at the time of random assignment. 
However, as mentioned earlier, for cost reasons, we chose only a sample of states for the UI 
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Firms often hire outside companies to report earnings to UI agencies. In these cases, the 
reporting of earnings may be more accurate. 
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study. In this section, we discuss our approach for selecting the 25 states for the UI study (22 of 
which provided us with wage records data), as well as the sample members included in the study. 

a. Selection of States 

Our power analysis indicated that the desired levels of precision on earnings impacts could 
be achieved if 17 states were selected for the UI study. The 17 states were randomly selected 
using systematic sampling techniques, where the probability a state was chosen was proportional 
to the number of 1993 Job Corps enrollees who lived in that state. Massachusetts and New York 
were excluded from the selection because we were informed that UI data could not be obtained 
from them. In addition, we did not include Alaska and Hawaii because these states (and Puerto 
Rico) were excluded from the National Job Corps Study. The states were grouped by region to 
ensure that the distribution of the chosen states would be dispersed throughout the United States. 
Regions 1 and 2 were grouped together, since these regions had a small number of enrollees after 
Massachusetts and New York were removed from the selection. 

We also selected eight “replacement” states for “primary” states that could not provide the 
UI data. We selected one replacement state for each region (regions 1 and 2 were grouped 
together) using the sampling techniques discussed above. Our design specified that a 
replacement state in a region would be contacted if UI data could not be obtained for any of the 
primary states in that region. 

During data collection, however, we did not follow this sequential design for obtaining UI 
data from replacement states, because of the time it took to determine whether primary states 
would provide their UI data for the study. For most states, it took some time to gain approval 
from the state UI offices to obtain their data and to have the UI offices match our sample 
members to their UI wage records. Because of the time lag in assessing whether the primary 
states would cooperate, we might not have been able to obtain data from the replacement states 
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in time for our report if we had waited until firm refusals were obtained from the primary states. 
Thus, we contacted the replacement states at the same time that we contacted the primary states. 
In other words, we attempted to collect data from all 25 selected states simultaneously rather 
than from the primary states first and from the replacement states second. Thus, our revised 
design does not distinguish between primary and replacement states. Instead, we treat all 25 
selected states as primary states. 

The following 25 states were selected for UI data collection: Arkansas, Arizona, California, 
Florida, Georgia, Idaho, Illinois, Kansas, Louisiana, Maryland, Maine, Michigan, Missouri, 
Mississippi, North Carolina, Nebraska, Nevada, New Jersey, Ohio, Oklahoma, Pennsylvania, 
South Carolina, Texas, Virginia, and Washington. Appendix A provides a detailed discussion of 
the state selection process and the selection results. 

b. State Response Rates 

Of the 25 states contacted about providing UI wage records, 22 agreed to provide the data. 
Only Georgia, Nevada, and Pennsylvania refused to release their data. 

We used several alternative procedures to adjust the sample weights to account for these 
three states. First, we assumed that only 22 states were originally sampled rather than 25. 
Second, we assumed that the nonresponding states were represented by the states in their region. 
These procedures produced similar results; we present the results using the first adjustment 
procedure (see Appendix A). 

c. Sample Members Included in the UI Study 

In the National Job Corps Study, all applicants during the sample intake period were asked 
to sign a consent form to access, collect, and use information for the study from records collected 
by public agencies, such as public assistance programs (AFDC/TANF, Medicaid, and the Food 
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Stamp Program), the UI program, child- support enforcement, and the criminal justice system. 
Applicants who refused consent for this data collection but agreed to participate in the full study 
were still allowed to enroll in Job Corps and were randomly assigned in the same way as other 
applicants. 

Many UI state agencies will release individual UI wage records only for those who provide 
written consent. Thus, only youths in the research sample who signed the consent form for 
records release were included in the UI study. As Table II.2 shows, about 79 percent of the 
research sample signed the consent form (78 percent for the control group and 79 percent for the 
program group). 

Signee rates were similar by gender, age, education level, and whether the youth received 
public assistance at program application (see Tables II.2 and A. 4). However, the rates differed 
substantially across regions (ranging from about 52 percent in region 1 to 93 percent in region 4), 
and by conviction history at program application (74 percent for those ever convicted, compared 
to 80 percent for those never convicted). Furthermore, a joint statistical test of the hypothesis 
that the distribution of baseline characteristics of signers and the full sample of signers and 
nonsigners are similar was rejected at the 1 percent significance level. Finally, and most 
important, we found using the SER data that the earnings impacts in 1999 to 2001 are 
substantially larger for signers than for nonsigners. Consequently, the UTbased impact estimates 
calculated using the sample of signers only may not be representative of Ul-based impacts for 
the full study population (that is, they may be subject to nonresponse bias). To help alleviate this 
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Signee rates were also somewhat higher for 48-month interview respondents than 
nonrespondents (77.3 percent, compared to 74.6 percent). 
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TABLE II.2 



PERCENTAGE OF SAMPLE MEMBERS WHO SIGNED THE RECORDS RELEASE CONSENT FORM, 
BY RESEARCH STATUS AND KEY SUBGROUP 



Subgroup 


Program Group 


Control Group 


Combined Sample 


Full Sample 


79.4 


78.3 


78.9 


Gender 


Male 


79.5 


78.7 


79.1 


Female 


79.4 


77.7 


78.6 


Age at Application 


16 to 17 


78.5 


77.7 


78.1 


18 to 19 


80.4 


79.5 


79.9 


20 to 21 


80.7 


78.2 


79.5 


22 to 24 


78.0 


77.4 


77.7 


Race/Ethnicity 


White, non-Hispanic 


81.9 


80.8 


81.4 


Black, non-Hispanic 


78.4 


77.3 


77.9 


Hispanic 


76.5 


77.4 


76.9 


Other 


81.0 


79.4 


80.2 


Region 


1 


53.5 


50.9 


52.2 


2 


62.8 


59.1 


60.8 


3 


56.5 


55.7 


56.1 


4 


92.6 


93.1 


92.8 


5 


85.3 


82.8 


84.0 


6 


87.4 


88.5 


87.9 


7/8 


76.9 


77.0 


76.9 


9 


84.1 


80.4 


82.2 


10 


86.6 


83.8 


85.2 


Education Level at Application 


Completed 12th grade 


79.6 


78.7 


79.2 


Did not complete 12th grade 


79.5 


78.7 


79.1 


Receipt of Public Assistance 


Received assistance 


80.0 


79.1 


79.5 


Did not receive assistance 


79.4 


78.8 


79.1 


Conviction History at Application 


Ever convicted or adjudged delinquent 


76.1 


72.1 


74.1 


Never convicted or adjudged delinquent 


79.7 


79.2 


79.5 


Residential Designation Status 


Resident 


79.6 


78.9 


79.3 


Nonresident 


78.6 


74.3 


76.5 


Sample Size 


9,369 


5,940 


15,309 



Source: Records release consent forms and ETA-652 and ETA-652 Supplement Data. 

Notes: All figures were calculated using sample weights to adjust for the sample design. Figures exclude 77 cases (37 

control group and 40 program group members) who were determined to have enrolled in Joh Corps prior to random 
assignment and were thus ineligible for the study. 
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problem, we used propensity scoring methods to adjust the sample weights for nonresponse to 
the consent form (see Appendix A). 

Importantly, because signatures on the consent form were requested prior to random 
assignment, there are very few differences in the baseline characteristics of program and control 
group signers (Tables II.2 and A.4). The joint test of the hypothesis that the distributions of 
baseline characteristics between the two groups of signers are similar is not statistically 
significant. Thus, although we find differences between the characteristics of signers and 
nonsigners, the characteristics of signers in the two research groups appear to be similar. 

The SSNs of all program and control group members who signed the consent form were sent 
to each of the 22 state UI agencies. Later in this section, we discuss our analytic approach for 
estimating UTbased earnings impacts accounting for the fact that not all sample members were 
in these 22 states during the follow-up period (which causes the UI earnings data to contain too 
many “zeroes”). 

3. Outcome Measures 

The UI data received from the 22 states contain quarterly earnings data for each reported job 
the sample members held from the first quarter in 1999 to the fourth quarter in 2001. The data 
cover this time period, because in most states these were the wage records that were available 
online when the UI agencies processed our data request. 

For each state and calendar quarter between 1999 and 2001, we constructed total quarterly 
earnings for each sample member by summing reported earnings from each of the youth’s 
employers (and put them into 1995 dollars using the GDP price deflator). A small number of 
outliers were set to missing (seven cases for the 1999 data, four for the 2000 data, and five for 
the 2000 data). 
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We also constructed quarterly UI earnings measures in random assignment time. As 
discussed, the use of random assignment time has the advantage that outcomes are measured 
after a fixed period since sample members were determined eligible for Job Corps. This feature 
eases the interpretation of the impact estimates. However, because a particular point after 
random assignment falls on a different calendar date across sample members, the specific 
quarterly measures in random assignment time that could be constructed varied across sample 
members. For example, the first quarter of 1999 was quarter 17 in random assignment time for a 
youth randomly assigned in April 1995, but it was quarter 15 for a youth randomly assigned in 
October 1995.^^ 

Table II. I displays sample sizes for the quarterly earnings measures constructed using the UI 
data. The figures pertain to those who lived in the 22 selected states at application to Job Corps 
(the effective sample size for the UI study). The effective sample sizes for the 1999, 2000, and 
2001 calendar-time measures are 4,613 program group and 2,855 control group members. Data 
for the measures in random assignment time are available for the full UI sample in quarters 17 to 
23, for most of the sample in quarters 15 to 16 and 24 to 26, but for only a small number of 
sample members in other quarters. Thus, we present UTbased impact estimates only for quarters 
15 to 26 (corresponding to years 5 and 6 and the first half of year 7 after random assignment). 

Because of small effective sample sizes for some population subgroups, the UI analysis 
focused only on estimating impacts for the full sample. 



'^'5 

We defined quarter 1 in random assignment time as the calendar quarter when the youth 
was randomly assigned if the random assignment date was in the first half of the calendar 
quarter, and as the calendar quarter after the youth was randomly assigned if the random 
assignment date was in the second half of the calendar quarter. For example, if a youth was 
randomly assigned on November 14, 1995, then quarter 1 in random assignment time was the 
fourth quarter in 1995, but if the youth was randomly assigned on November 16, 1995, then 
quarter 1 in random assignment time was the first quarter of 1996. 
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4. Analysis Samples 

To analyze the UI data, we used two analysis samples of those who signed the consent form 
for records release. First, we analyzed UI data on the full sample of signers in the research 
sample to obtain nationally representative Ul-based earnings impact estimates. As we discuss in 
the next section, we used this sample to (1) assess whether the UTbased and survey-based impact 
estimates differ due to the combined effect of survey nonresponse bias and reporting differences 
between the two data sources, and (2) obtain impact estimates covering the postsurvey period 
(that is, in years 5 and 6 and the first half of year 7 after random assignment). 

The second analysis sample includes signers who (1) completed 48-month interviews, and 
(2) reported in the interviews that they lived in the 22 selected states for the entire four-year 
follow-up period and did not work in other states. Complete UI earnings information is available 
for these youths. Thus, UTbased and survey-based earnings and employment measures were 
directly compared for individuals in this sample. Consequently, we used this sample to assess the 
extent to which earnings impact estimates using the two data sources differ due to reporting 
differences. This sample, however, could not be used to assess the extent of survey nonresponse 
bias. 

5. Analytic Methods 

To produce unbiased UTbased impact estimates that can be generalized to the study 
population for the National Job Corps Study, several important issues need to be considered. 
First, because we selected only a sample of states for the UI study, we cannot determine the 
exact UTbased earnings for all sample members during the follow-up period, because the youths 
may have had UTbased earnings from states out of the sample. A sample member with a UI 
record from the selected states clearly had some UTbased earnings during the follow-up period. 
However, the youth may have had other UTbased earnings from a nonselected state if the youth 
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changed states during the follow-up period (thereby causing the UI data to underestimate the 
individual’s total earnings). Similarly, a sample member with no UI record from the selected 
states may have had Ul-based earnings from a nonselected state. 

A second issue is that we need to account for unequal weighting due to the sampling of 
states. In addition, the weights need to account for the three states that would not release their 
records data, as well as for those sample members who did not sign the records release consent 
form when they applied to Job Corps (see Appendix A). 

As discussed, we conducted the UI analysis using two samples. First, we used the full 
analysis sample. Second, we used only sample members who completed 48-month follow-up 
interviews and who reported living in the selected states during the entire four-year follow-up 
period. Next, we discuss the analytic methods that we used to obtain unbiased Ul-based earnings 
impact estimates using these two samples. 

a. Analysis Using the Full Sample 

The states selected for UI records collection were chosen randomly. In principle, the random 
selection of states ensures that the expected number of sample members who moved into the 
selected states during the follow-up period should be equal to the expected number of sample 
members who moved out of the selected states during that time. Thus, the expected number of 
sample members who resided in the selected states should have remained constant over time, 
although the distribution of specific sample members residing in these states may have changed. 

We tested the accuracy of this “steady-state” assumption using survey information on the 
states in which sample members lived and worked during the four-year follow-up period. 
Table II.3 displays this mobility information for those who lived in the 22 UI states at program 
application and those who lived in the other (non-UI) states. We present the figures separately 
for the program and control groups and for males and females. 
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TABLE II.3 



MOBILITY ACROSS STATES, BY RESEARCH STATUS AND GENDER 





Eull Sample 


Males 




Eemales 


Mobility Measure 


Program 

Group 


Control 

Group 


Program 

Group 


Control 

Group 


Program 

Group 


Control 

Group 


Mobility Across All States 

Percentage Ever Lived or Worked in a 
Different State to the State Lived in at 
Application to Job Corps 














Years 1 to 4 


Those in all states at application 


34.3 


28.9 


38.3 


33.3 


28.8 


22.4 


Those in the 22 UI states 


33.0 


27.3 


37.0 


31.9 


27.0 


20.7 


Those in other states 


36.7 


31.7 


40.3 


35.6 


31.6 


25.7 


Years 3 and 4 


Those in all states at application 


26.5 


24.3 


29.6 


27.9 


22.1 


19.1 


Those in the 22 UI states 


24.9 


22.3 


28.1 


26.1 


20.1 


16.9 


Those in other states 


29.3 


27.8 


32.0 


30.9 


25.4 


23.0 


Years 3 and 4 for at least 6 months 


Those in all states at application 


19.8 


18.2 


21.4 


20.3 


17.4 


15.3 


Those in the 22 UI states 


18.2 


17.0 


20.2 


19.4 


15.3 


13.7 


Those in other states 

Mobility Across tbe UI and Non-UI 
States 

Weighted Number of Those in the UI 
States at Application Who Lived or 
Worked in a Non-UI State“ 


22.4 


20.3 


23.5 


21.7 


20.8 


18.1 


Years I to 4 


9,334 


6,850 


6,231 


4,747 


3,103 


2,103 


Years 3 and 4 


6,433 


5,264 


4,303 


3,721 


2,130 


1,543 


Years 3 and 4 for at least 6 months 

Weighted Number of Those in the 
Non-UI States at Application Who 
Lived or Worked in a UI State'* 


4,292 


3,715 


2,791 


2,449 


1,502 


1,266 


Years I to 4 


7,618 


6,856 


4,889 


4,601 


2,730 


2,256 


Years 3 and 4 


6,019 


5,750 


3,803 


3,849 


2,216 


1,901 


Years 3 and 4 for at least 6 months 


4,220 


4,136 


2,503 


2,603 


1,717 


1,533 


Unweighted Sample Size 


6,828 


4,485 


3,741 


2,787 


3,087 


1,698 



Source: ETA-652 data and baseline, 12-, 30-, 48-month follow-up interview data for those who completed 48-month 

interviews. 

Note: All figures were calculated using sample weights to account for the sample and survey designs and interview 

nonresponse. 

'“About 50,500 (or 62.5 percent) of the 80,883 youths in the sample universe for the evaluation lived in the 22 UI states at 
application to Job Corps. 
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The steady-state assumption is supported by the data. About one-third of the sample ever 
lived or worked in a state different from the one in which they lived at baseline, and the figure 
was about one-quarter during years 3 and 4 after random assignment (the postprogram period). 
Mobility rates were somewhat lower for control group than program group members, and for 
those in the 22 UI states than those in the non-UI states. However, importantly, the number of 
youths in the sample universe of 80,883 who initially lived in the 22 UI states but who moved to 
the non-UI states was similar to the number of youths who moved in the reverse direction. This 
result holds for both research groups and by gender. For example, in years 3 and 4 after random 
assignment, the program group data indicate that 6,433 eligible applicants in the sample universe 
moved into the 22 states, whereas 6,019 moved into the non-UI states. Furthermore, key 
baseline characteristics (such as age, education level, application date, and arrest history) of 
those who moved into the 22 UI states are similar to those who moved into the non-UI states (not 
shown) and the survey-based earnings are somewhat similar for the two groups (about $11,000 
in year 4 after random assignment for those who moved out of the UI states, compared to 
$10,000 for those who moved into the UI states). 

Under the steady-state assumption, the mean outcome for the program group in a particular 
state can be estimated by summing the reported earnings for all program group members in that 
state and dividing this sum by the number of program group members who lived in that state at 
application to Job Corps, and similarly for the control group. The estimated impact per eligible 
applicant for a state is then obtained as the difference between the program and control group 



24 

The figures for the program group in years 1 to 4 are much larger, because in years 1 and 
2, many program group members enrolled in a Job Corps center in a different state than the one 
from which they came. 
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means in that state, and the estimated impact for the study population is a weighted average of 
these state impacts (see Appendix A for more details on this estimation procedure). 

The intuition behind this estimation procedure can be demonstrated using an example. 
Suppose a youth who initially lived in California moved out of California one year after random 
assignment. Then, the estimation procedure relies on the steady-state assumption that the 
earnings of this youth after leaving California can be represented by the earnings of another 
youth who initially lived outside California but moved there one year after random assignment. 

It is important to reiterate that this estimation procedure cannot be used to compare survey- 
based and Ul-based measures for individual sample members but can be used only to estimate 
aggregate Ul-based impacts. The procedure cannot be used to obtain separate impact estimates 
for survey respondents and nonrespondents and hence cannot directly be used to assess the extent 
of interview nonresponse bias. Nor can the procedure be used to directly assess reporting 
differences between the two data sources. Instead, the procedure produces estimates that reflect 
the joint effects of interview nonresponse bias and reporting differences — the relative 
contribution of each factor cannot be disentangled. 

b. Analysis Using Those Who Completed the 48-Month Follow-Up Interview 

We can directly compare employment and earnings measures from the UI and survey data 
using the sample of youths who completed the 48-month interview and who reported in the 
interviews that they lived in the selected states during the entire four- year follow-up period and 
did not work in outside states. Complete UI data are available for this sample, because they had 
zero earnings in the states not selected for UI data collection. Thus, we can use this sample to 
assess the extent of reporting differences between the UI and survey data. 

For this analysis, the Ul-based earnings measures for each person were constructed by 
summing the person’s earnings across the 22 selected states (that is, we constructed total UT 
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based earnings from all the states combined). We then compared the Ul-based and survey-based 



earnings measures at the individual and aggregate level, and the estimated earnings impacts 
using the two data sources. 
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III. IMPACT FINDINGS 



This chapter presents impact findings using the SER and UI data and compares them to 
impact findings using the survey data. We present estimated impacts per eligible applicant for 
employment and earnings outcomes measured in calendar time (for all three data sources) and 
random assignment time (for the survey and UI data). We present results in both calendar and 
random assignment time to maximize the sample available for the analysis. To roughly link the 
two sets of outcome measures, one can view 1996 as year 1 after random assignment for most 
sample members (and part of year 2 for some) and, thus, 2001 as year 6 for most (and part of 
year 7 for some). 

In this chapter, we first present administrative-based impact results for the full sample and 
for 48-month interview respondents only. Impacts are presented for the four- year period covered 
by the survey and afterwards. Second, we present impact results for key population subgroups. 

A. IMPACT FINDINGS FOR THE FULL SAMPLE AND FOR 48-MONTH 

INTERVIEW RESPONDENTS 

Tables III.l and III.2 present impact findings for the full sample. Table III.l presents 
estimated impacts per eligible applicant on calendar-yeax earnings (in 1995 dollars) and 
calendar-year employment rates using the SER, UI, and survey data. The table displays means 
and percentages for the program and control groups, differences between these means and 
percentages (that is, estimated impacts per eligible applicant) and the statistical significance of 
the estimated impacts. Some table entries are missing for the survey and UI data, because, as 
discussed, calendar-year measures for some years could not be constructed using these data. 
Table III.2 displays similar statistics for quarterly earnings (in 1995 dollars) and quarterly 
employment rates measured in random assignment time that were obtained using the survey and 
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IMPACTS ON CALENDAR YEAR EARNINGS AND EMPLOYMENT RATES EOR THE EULL SAMPLE, 
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Sample Size 6,828 4,485 11,313 9,264 5,874 15,138 4,613 2,855 7,468 




Sources: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; (2) annual social security earnings 

records for the full research sample; and (3) quarterly UI earnings records from the following 22 randomly selected states for those who signed the 
records release consent form: AR, AZ, CA, FL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, TX, VA, and WA. 
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TABLE III.2 



IMPACTS ON EARNINGS AND EMPLOYMENT USING SURVEY AND UI DATA 
FOR THE FULL SAMPLE, BY QUARTER AFTER RANDOM ASSIGNMENT 



Outcome Measure 






Data Source 








Survey Data 


Quarterly UI Earnings Records 
from 22 States 


Program 

Group 


Control 

Group 


Estimated 

Impact* 


Program 

Group 


Control 

Group 


Estimated 

Impact* 


Average Earnings, by Quarter 














After Random Assignment (in 














1995 Dollars) 














1 


565.4 


851.4 


-286.0*** 








4 


1,201.0 


1,378.4 


-177 4*** 








8 


1,992.8 


1,909.8 


83.0* 








12 


2,550.7 


2,321.5 


229.2*** 








13 


2,669.4 


2,444.1 


225.4*** 








14 


2,727.6 


2,524.2 


203.5*** 








15 


2,778.3 


2,564.2 


214 1*** 


1,396.5 


1,299.1 


97.3 


16 


2,827.0 


2,591.6 


235.4*** 


1,414.8 


1,382.0 


32.8 


17 








1,449.7 


1,470.9 


-21.2 


18 








1,508.9 


1,511.6 


-2.7 


19 








1,545.6 


1,553.5 


-7.9 


20 








1,568.6 


1,593.0 


-24.4 


21 








1,632.6 


1,677.3 


-44.8 


22 








1,707.8 


1,772.0 


-64.3 


23 








1,721.7 


1,775.5 


-53.8 


24 








1,800.4 


1,857.7 


-57.3 


25 








1,856.2 


1,909.0 


-52.8 


26 








1,909.0 


1,955.6 


-46.6 


Percentage Employed, by Quarter 














After Random Assignment 














1 


33.2 


42.1 


_§ 9*** 








4 


49.8 


57.7 










8 


59.0 


57.9 


1.2 








12 


66.2 


63.0 


2 2*** 








13 


66.8 


63.4 










14 


67.5 


65.1 


2 4*** 








15 


69.2 


65.6 


3.6*** 


55.0 


52.6 


2.3 


16 


71.1 


68.7 


2 4*** 


55.4 


55.1 


0.3 


17 








55.2 


54.9 


0.3 


18 








54.9 


55.9 


-1.0 


19 








55.2 


56.3 


-1.1 


20 








53.6 


55.5 


-1.8 


21 








53.9 


55.5 


-1.6 


22 








55.3 


57.1 


-1.7 


23 








55.5 


57.4 


-2.0 


24 








55.3 


58.5 


-3.3** 


25 








56.5 


58.6 


-2.1 


26 








58.1 


61.0 


-2.8 


Sample Size 


6,828 


4,485 


11,313 


4,613 


2,855 


7,468 



Sources: (I) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; and 

(2) quarterly UI earnings records from the following 22 randomly selected states for those who signed the records 
release consent form: AR, AZ, CA, EL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, TX, 
VA, and WA. 
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TABLE III. 2 {continued) 



Notes: 1. Blank entries signify that figures are not applicable because data were not available or sample sizes were too 

small to generate precise estimates. 

2. All estimates were calculated using sample weights to account for (1) the sample design (for both data sources), 
(2) the survey design and interview nonresponse (for the survey data) and (3) the selection of states to the UI 
sample and nonresponse to the records release form (for the UI data). Standard errors of the estimates account 
for design effects due to the unequal weighting of the data and clustering of areas for in-person interviews at 
baseline (for the survey data) and the selection of states (for the UI data). 

■“These estimated impacts pertain to eligible applicants, and are measured as the difference between the weighted means for 
program and control group members. 

*Significantly different from zero at the .10 level, two-tailed test. 

**Significantly different from zero at the .05 level, two-tailed test. 

***Significantly different from zero at the .01 level, two-tailed test. 
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UI data. Again, some table entries pertaining to particular quarters are missing when data were 
not available to construct outcome measures pertaining to these quarters. 

Tables III.3, III.4, and B.2 present results from our analysis examining the extent to which 
differences between the findings using the survey and administrative data are due to survey 
nonresponse and due to reporting differences between the two data sources. Table III. 3 displays 
impact results using the SER data on calendar year earnings for respondents to the 48-month 
interview (along with the full sample results), and Table B.2 displays impact results for survey 
nonrespondents. Table III.4 compares UTbased and survey-based earnings impacts for survey 
respondents who lived in the 22 states selected for the UI study for the entire four- year follow-up 
period and who did not work elsewhere. We have complete UI earnings data for this sample, so 
differences in the survey-based and UTbased earnings can be attributed solely to reporting 
differences between the two data sources. 

Before presenting the results, it is important to recognize that differences in earnings levels 
using the administrative records and survey data will lead to differences in earnings impacts in 
nominal (or dollar) terms using the two data sources. However, if the ratios of survey-based to 
administrative-based earnings levels are similar for the program and control groups, then the 
earnings impacts relative to the control group means (that is, in percentage terms) will be similar 
using the two data sources. The percentage earnings gains, however, will differ if the earnings 
ratios differ for the program and control groups. 

To illustrate these points, the earnings impact using the survey data, 1$, can be written as 
follows: 

( 1 ) Is = EpaRp - EcaR c. 
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IMPACTS ON CALENDAR YEAR EARNINGS AND EMPLOYMENT RATES EOR RESPONDENTS 
TO THE 48-MONTH INTERVIEW, BY DATA SOURCE 
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Sample Size 6,828 4,485 11,313 6,772 4,451 11,223 9,264 5,874 15,138 
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TABLE III.4 



IMPACTS ON EARNINGS AND EMPLOYMENT USING SURVEY AND UI DATA 
EOR SURVEY RESPONDENTS WHO LIVED IN THE UI STATES 
EOR THE ENTIRE 48-MONTH EOLLOW-UP PERIOD, 

BY QUARTER AETER RANDOM ASSIGNMENT 



Data Source 










Quarterly UI Earnings Records 






Survey Data 






from 22 States 






Program 


Control 


Estimated 


Program 


Control 


Estimated 


Outcome Measure 


Group 


Group 


ImpacU 


Group 


Group 


ImpacL 


Average Earnings, by 
Quarter After Random 
Assignment (in 1995 
Dollars) 














1 


577.0 


856.0 


-279.0*** 








4 


1,191.7 


1,369.1 


-177 4*** 








8 


1,952.4 


1,839.5 


112.9 








12 


2,392.3 


2,273.1 


119.1 








13 


2,517.0 


2,407.1 


109.8 








14 


2,574.2 


2,469.5 


104.7 








15 


2,642.1 


2,466.1 


175.9** 


1,296.5 


1,309.9 


-13.5 


16 


2,695.6 


2,507.6 


188.0** 


1,409.4 


1,381.8 


27.6 


17 








1,454.1 


1,410.3 


43.8 


18 








1,449.1 


1,458.7 


-9.5 


19 








1,457.2 


1,476.0 


-18.8 


20 








1,523.8 


1,510.6 


13.2 


21 








1,564.6 


1,608.5 


-43.9 


22 








1,626.5 


1,660.4 


-33.9 


23 








1,704.6 


1,678.5 


26.1 


24 








1,763.7 


1,739.3 


24.5 


25 








1,841.3 


1,825.8 


15.5 


26 








1,897.7 


1,824.2 


73.5 


Percentage Employed, 
by Quarter After 
Random Assignment 














1 


34.3 


42.8 


-8.5*** 








4 


51.0 


56.6 


-5.6*** 








8 


58.7 


57.6 


1.1 








12 


64.6 


63.5 


1.1 








13 


65.7 


63.7 


1.9 








14 


66.1 


64.6 


1.4 








15 


68.3 


64.7 


3.6** 


55.0 


55.8 


-0.7 


16 


70.8 


69.1 


1.8 


58.2 


55.2 


3.0* 


17 








57.7 


55.8 


1.9 


18 








56.5 


57.4 


-0.9 


19 








56.6 


57.3 


-0.8 


20 








55.6 


56.0 


-0.3 


21 








56.2 


56.0 


0.2 


22 








56.7 


57.5 


-0.8 


23 








58.4 


57.9 


0.5 


24 








57.9 


57.8 


0.1 


25 








58.7 


58.2 


0.5 


26 








58.6 


58.5 


0.1 


Sample Size*’ 


2,754 


1,872 


4,626 


2,754 


1,872 


4,626 
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TABLE III.4 {continued) 



(1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; and 

(2) quarterly UI earnings records from the following 22 randomly selected states for those who signed the records 
release consent form: AR, AZ, CA, FL, ID, IL, KS, LA, MD, ME, MI, MO, MS, NC, NE, NJ, OH, OK, SC, TX, 
VA, and WA. 

1. Blank entries signify that figures are not applicable because data were not available or sample sizes were too 
small to generate precise estimates. 

2. All estimates were calculated using sample weights to account for (1) the sample design (for both data sources), 
(2) the survey design and interview nonresponse (for the survey data). The estimates do not account for the 
selection of states to the UI sample. Standard errors of the estimates account for design effects due to the unequal 
weighting of the data and clustering of areas for in-person interviews at baseline. 

■“These estimated impacts pertain to eligible applicants, and are measured as the difference between the weighted means for 
program and control group members. 

’’ The sample includes those who (1) completed the 48-month interview, (2) signed the records release consent form, and (3) lived 
and worked only in the 22 states selected for the UI study. 



*Significantly different from zero at the .10 level, two-tailed test. 

**Significantly different from zero at the .05 level, two-tailed test. 

***Significantly different from zero at the .01 level, two-tailed test. 



Sources: 



Notes: 
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where Epa is average administrative-based earnings for the program group, Rp is the ratio of 
survey-to-administrative average earnings for the program group, and Eqa and Rc are the 
corresponding values for the control group, respectively. If the reporting ratios are similar for the 
program and control groups (that is, if Rp = Rc = R) , then the survey-based impact in equation 
(1) reduces to (Epa - Eca)R, and the control group mean using the survey data becomes EcaR- In 
this case, the survey-based impact estimate, (Epa - Eca)R, will differ from the administrative- 
based impact estimate, (Epa - Eca), by the reporting factor R, but the two-sets of impacts relative 
to their control group means will be the same (that is, they both can be expressed as [(Epa - 
Eca)/Eca\ ■ Clearly, this argument does not hold if the reporting ratios differ by research status 
(that is, if Rp differs from Rc). 

1. Impact Findings During the Period Before Random Assignment 

As demonstrated in Schochet (1998), the random assignment process generated program and 
control groups with similar observable characteristics in the period prior to random assignment. 
This analysis was conducted using demographic data from Job Corps intake (ETA-652) forms 
(which were collected at the time of program application and are available for all sample 
members) and detailed baseline interview data (which are available for 93 percent of sample 
members who completed the baseline interview). 

The 1993 and 1994 SER data were used to further check that the random assignment process 
was implemented correctly. Random assignment started in mid-November 1994, and thus 1993 
is a pre-random- assignment period for all sample members. Calendar year 1994 is a pre- 
random-assignment period for the 96.5 percent of sample members who were randomly assigned 
in 1995 or 1996, and is largely a pre-random-assignment period for the 3.5 percent of sample 
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members who were randomly assigned in late 1994. Consequently, we expect differences in 
the average SER earnings of program and control group members in 1993 and 1994 to be 
statistically insignificant. 

The data support these expectations. Earnings differences between the program and control 
groups in 1993 and 1994 are small and statistically insignificant at the 10 percent level for the 
full sample and for all key population subgroups except for male nonresidents (Table III.l and 
Tables B.3 to B.16). 

Annual earnings levels and employment rates were low in these years, because many sample 
members were in school then because of their young ages (in 1993, sample members were 
between ages 14 and 22). In 1993, only about 43 percent of sample members were employed, 
and the average sample member earned only about $1,000 (which translates into $2,325 per 
worker; Table III.l). In 1994, average earnings increased to only slightly more than $1,500. 

2. Impact Findings During the Four- Year Period Covered by the Survey 

In this section, we present impact findings during the four-year, post-random-assignment 
period covered by the survey. First, we present impact findings using the administrative data for 
the full sample of 48-month interview respondents and nonrespondents. Second, we present the 
administrative-based impact findings using interview respondents only. 



Among program group members who enrolled in Job Corps, the average time between 
random assignment and program enrollment was 1.4 months. Thus, many enrollees who were 
randomly assigned in 1994 did not enroll in Job Corps until 1995. Thus, 1994 can be considered 
a preprogram period for nearly all enrollees. 
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a. Impact Findings for the Full Sample of Survey Respondents and Nonrespondents 

The pattern of the estimated impacts using the survey and administrative data is similar in 
periods covered by both data sources. However, the survey-based impact estimates are larger 
and more often statistically significant (Table III.l). 

The estimated impacts on annual earnings in 1995 (according to the SER data) and in 1996 
(according to both the survey and SER data) are negative and statistically significant, because 
many program group members were enrolled in Job Corps then. The estimated earnings impacts 
became positive and statistically significant in 1997 according to both data sources as program 
group members left Job Corps and found jobs. In 1998, the earnings impacts remained positive 
and statistically significant for both data sources, although the survey-based impact estimate is 
larger ($972 or a 10.4 percent earnings gain relative to the control group mean using the survey 
data, compared to $220 or a 3.9 percent earnings gain relative to the control group mean using 
the SER data). Similarly, the estimated impact on the annual employment rate in 1998 is larger 
using the survey data (2.4 percentage points, compared to 1.3 percentage points). 

The impact estimates according to the survey data are also larger than those according to the 
UI data (Table III.2). While the survey-based impact estimates in quarters 15 and 16 after 
random assignment are positive and statistically significant, the corresponding Ul-based impact 
estimates are smaller and not statistically significant. Similarly, the employment rate impact in 
quarters 15 and 16 are statistically significant using the survey data but not using the UI data. 



The estimated impacts on the 1995 and 1996 employment rates based on the SER data are 
large and positive, because wages that Job Corps pays to enrollees (that is, student pay) are 
reported to the IRS. Thus, the reported employment rates for the program group were very high 
during these in-program years, because most Job Corps enrollees receive student pay. 
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The estimated impacts in nominal (or dollar) terms are larger according to the survey than 
the administrative records data, because average reported earnings levels are substantially higher 
in the survey data for both the program and control groups. For example, in 1998, average 
earnings for the program group are about $10,300 (or $12,650 per worker) using the survey data 
but only $5,800 (or $6,850 per worker) using the SER data. 

The impacts in percentage terms are also somewhat larger using the survey data during the 
postprogram period, because percentage differences in earnings levels according to the survey 
and administrative data are slightly larger for the program than the control group. For example, 
average reported calendar-year earnings in 1998 are about 77 percent higher according to the 
survey data than the SER data for the program group, whereas the corresponding figure for the 
control group is 67 percent (Table III.l). Similarly, average quarterly earnings in quarter 16 after 
random assignment are twice as large using the survey data than the UI data for the program 
group, but are only 88 percent larger using the survey data for the control group (Table III.2). 

It is important to emphasize that differences in the survey-based and administrative-based 
earnings levels are due to the combined effects of survey nonresponse and reporting differences 
between the two types of data sources. We attempt to isolate each of these effects in the next 
section, where we compare earnings levels as measured by the survey and administrative data 
using comparable samples of survey respondents only. 

Finally, average earnings are about 17 percent higher according to the SER data than the UI 
data. As discussed in Chapter II, coverage is similar for the UI and OASDI programs, and both 
programs cover nearly all formal jobs. Differences in the earnings levels in the SER and UI data 

97 

We find a similar result using the SER data for only those who signed the records release 
consent form (so that the UTbased and SER-based estimates are directly comparable). 
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probably reflect the lower accuracy of the UI data, because, as discussed (1) employers have 
financial incentives to underreport earnings to state UI programs but to fully report earnings to 
the IRS; (2) the SSA verifies SSNs before inputting reported earnings data or matching people to 
their earnings records, whereas UI agencies do not; and (3) the UI data contain information on 
only the 80 percent of sample members who signed the records release consent form, whereas 
the SER data pertain to all sample members. The 17 percent SER-to-UI average earnings ratio is 
similar to the ratios reported for youths in the National JTPA Study (Kornfeld and Bloom 1999). 

b. Impact Findings for 48-Month Survey Respondents Only 

As discussed, the percentage earnings gains in 1998 are somewhat larger using the survey 
than using SER data. In this section, we address the following question: To what extent are the 
larger percentage earnings gains in the survey than administrative data due to survey 

nonresponse, and to what extent are they due to reporting differences between the two data 

28 

sources that are correlated with research status? 

To help disentangle these effects, we present earnings impacts using the administrative 
records data /or the sample of 48-month survey respondents only, so that the impact estimates 
using the survey and administrative data can be compared directly. We also discuss differences 
in earnings levels as measured by the survey and administrative records data using comparable 
samples of interview respondents. 

Earnings Impact Findings. We find that differences in the percentage earnings gains in 
1998 using the survey and SER data are due in roughly equal parts to interview nonresponse bias 
and to reporting differences between the two data sources (Table III.3). The estimated 1998 
earnings gain is 10.4 percent according to the survey data and 3.9 percent according to the SER 
The percentage earnings gains in 1997, however, are similar using the two data sources. 
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data. Using the sample of respondents to the 48-month interview only, the SER-based earnings 
gain increases from 3.9 to 6.9 percent, which is still smaller than the 10.4 percent survey-based 
figure. Thus, the remaining difference between the survey and full-sample SER impacts is due to 
reporting differences between the two data sources that are slightly greater for the program than 
control group. 

To quantify the contribution of each factor, we can decompose the difference between the 
survey-based and full-sample SER-based earnings impacts (in percentage terms) as follows: 

(2) (Sr-A) = (S-A) + (Sr-S), 

where Sr is the survey-based impact (percentage earnings gain) for survey respondents, A is the 
administrative-based impact for the full sample, and S is the survey-based impact if all sample 
members had responded to the survey. The expression (S-A), then, represents the extent of 
reporting differences between the two data sources, and (Sr-S) represents the extent of survey 
nonresponse bias. Equation (2) is operational if we assume that (S-A) can be estimated by (Sr- 
Ar), and that (Sr-S) can be estimated by (Ar-A), where Ar is the administrative-based impact for 
survey respondents. 

Using equation (2), we estimate that, in 1998, about 46 percent of the difference between the 
survey-based and full sample SER-based earnings impacts (in percentage terms) is due to 
interview nonresponse bias, and 54 percent is due to reporting differences between the two data 
sources. 

The interview nonresponse bias is caused by larger impacts for survey respondents than 
nonrespondents, which suggests that the survey-based earnings impact estimates are somewhat 
biased upward (Table B.2). The interview nonresponse bias can be attributed to two potential 
sources. Eirst, it could be due to differences between the baseline characteristics of respondents 
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in the program and control groups that are correlated with earnings. If interview respondents in 
the program group were drawn from a somewhat more advantaged subpopulation of the full 
program group than was true for interview respondents in the control group, then the survey- 
based impact estimates would be biased upward. Second, it is possible that respondents in the 
program and control groups are similar but that earnings impacts are actually larger for survey 
respondents than for survey nonrespondents. In this case, the impacts based on survey 
respondents are unbiased, but they are not generalizable to the full study population. As 
discussed in Chapter IV, we believe that both factors account for the interview nonresponse bias, 
although the latter factor is more important. 

Results using the UI data provide additional evidence on reporting differences between the 
survey and administrative data that are larger for the program than control group. Using the 
sample of survey respondents who lived in the 22 UI states for the entire four-year follow-up 
period and who did not work elsewhere, we find that the survey-based earnings impacts in 
percentage terms are larger than the Ul-based ones in quarters 15 and 16 after random 
assignment (Table III.4). For example, in quarter 16, the survey data yield a statistically 
significant earnings gain of 7.1 percent, whereas the UI data yield a statistically insignificant 
earnings gain of only 2.0 percent. 

Differences in Earnings Levels, by Data Source. We find that average earnings levels 
continue to be much larger using the survey than administrative records data using the survey 
sample only. Average reported earnings levels in the survey data are about 70 percent higher 
than for survey respondents in the SER data. Similarly, average survey-based earnings are about 
80 to 90 percent higher than average UTbased earnings using a comparable sample. As discussed 
in Chapter IV, these reporting differences may be due to earnings from informal (cash-only) jobs 
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or from formal jobs not covered in the administrative records data. Alternatively, they may be 
due to survey respondents reporting more earnings to interviewers than they actually earned. 

The differences in reporting levels found in the survey and administrative data are somewhat 
larger than those found in previous studies using similar populations. For example, Kornfeld and 
Bloom (1999) found that mean quarterly earnings were about 35 to 70 percent higher according 
to the survey data than the UI data for youths in the National JTPA Study. Similarly, Cave 
(1995) found that the survey to UI ratios ranged from about 1.05 to 1.80 from several studies that 
examined the earnings of welfare recipients in welfare-to-work demonstration programs. These 
studies also found that the extent of underreporting was similar for the program and control 
groups, so the impact estimates based on the administrative records and survey data were usually 
similar. 

One possible explanation for the lower reported earnings levels in the administrative data in 
our study than the National JTPA Study is that the National Job Corps Study was conducted 
during a period of stronger economic growth. The unemployment rate for the civilian population 
of those age 16 and older was about 7.0 in the early 1990s (when the JTPA evaluation was 
conducted) but decreased to 5.5 percent in late 1994, to 4.5 percent in mid- 1998, and to 4 percent 
in 2000 (although it increased to about 5 percent by the end of 2001). Similarly, the 
unemployment rate for those ages 16 to 19 decreased from 17 percent to under 14 percent during 
the same period. In addition, inflation was low. Thus, in the tight labor market, our sample 
members may have been more likely to collect “under-the-table” earnings from casual or cash- 
only jobs that are reported in the survey but not in the administrative data. In addition, because 
the program group had higher skills than the control group (as measured by higher GED 
completion rates, many more hours spent in academic classes and vocational training, and higher 



56 




functional literacy test seores [Sehoehet et al. 2001; and Glazerman et al. 2000]), the program 
group may have earned more from these informal jobs than the eontrol group. 

Interestingly, the ealendar year employment rates eomputed using the survey and 
administrative data are similar, and are even somewhat lower according to the survey data 
(Table III.3). These findings further support our hypothesis that earnings levels are higher in the 
survey data because sample members were employed in second jobs or earned additional wages 
on their jobs that were not reported to the government. 

Reported quarterly employment rates, however, are much higher in the survey data than the 
UI data (Table III.4), whieh suggests that the administrative data are not eapturing short-term 
informal jobs held by sample members. These findings also suggest that sample members 
reported in the interviews that they were employed on jobs longer than they aetually were 
(perhaps because of recall error), resulting in annual employment rates that are similar using the 
two data sourees but quarterly employment rates that are higher using the survey data. The lower 
quarterly employment rates in the UI data may also be due to errors in the matching process used 
to obtain the UI data, beeause of inaeeurate SSNs. 

3. Impact Findings After the Period Covered by the Survey 

Based on the administrative data, we find no impacts of Job Corps on employment or 
earnings after the four-year period eovered by the survey. The estimated impaets on ealendar 
year earnings in 1999, 2000, and 2001 (using the SER data) and on quarterly earnings in quarters 
17 to 26 after random assignment (using the UI data) are all near zero and not statistieally 
significant (Tables III.l and III.2). 

Interestingly, the earnings impaets in the postsurvey period using the survey respondents 
only are also not statistically significant (Tables III.3 and III.4). The SER-based impacts using 
the survey respondents are larger than the SER-based impaets using the full sample beeause of 
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interview nonresponse bias. However, the SER-based earnings gains in 1999, 2000, and 2001 
using the survey respondents are still very small (only about 2.5 percent) and are not statistically 
significant. 

As Figure III.l shows, the control group caught up to the program group in 1999. The mean 
earnings of the program group based on the SER data grew sharply between 1996 and 1998, but 
grew at a slower rate between 1999 and 2001. The mean earnings of the control group grew more 
uniformly throughout the follow-up period. 

These findings are surprising. In our benefit-cost analysis, we assumed, based on the 
available evidence from the survey data, that the dollar value of the annual earnings impact in the 
fourth year after random assignment would persist without decline for the rest of a youth’s 
working lifetime (McConnell and Glazerman 2001). This extrapolation assumption was made for 
four main reasons. First, the impacts of Job Corps did not decline during years 3 and 4. In long- 
term studies of the returns to training, if returns decline, the decline occurs within two or three 
years after a trainee leaves the program (Eillard and Tan 1992; and Ashenfelter 1978). Second, 
Job Corps teaches many skills. The employment and training programs that have been found to 
have long-term impacts tend to be ones that teach a wide range of skills, such as the National 
Supported Work Demonstration (Couch 1992) and the Center for Employment Training 
(Zambrowski and Gordon 1993). On the other hand, those that have short-lasting impacts, such 
as on-the-job training and other private- sector training (Eillard and Tan 1992) and classroom 
training under the Manpower Demonstration Training Act (Ashenfelter 1978) tend to teach 
specific skills. Third, we found that Job Corps improves literacy and numeracy skills, which we 
would expect to have long-lasting impacts. Finally, the 12 percent earnings gains in year 4 based 
on the survey data are similar to the returns to a year of school found in studies based on returns 
over the entire working lives of individuals (Card 1999). 
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Why, then, did the control group catch up? Although it is too early to conclude that these 
findings will be permanent, we surmise that the earnings impacts decreased as the gap between 
the job-specific skills of program and control group members closed as the youths gained work 
experience. Job Corps participants experienced an initial boost in their employment and earnings 
soon after they left the program (that is, in years 3 and 4 after random assignment). This boost 
was caused by increases in participants’ skill levels, as measured by (1) large impacts on hours 
spent in education and training (equivalent to about one full school year), (2) large impacts on 
the attainment of GED and vocational certificates (about 20 to 30 percentage points), and 
(3) small impacts on functional literacy skills assessments at 30 months. The boost also may 
have been due to job placement services Job Corps offered. During this initial postprogram 
period, however, the differences in employment rates between the program and control groups 
were small (about 2 to 3 percentage points per quarter in year 4). Furthermore, although some 
evidence exists that the program group obtained better jobs, the differences were not large; most 
workers in both groups held low-paying jobs with high turnover. For example, the survey data 
indicate that, in their most recent job in quarter 16, workers in the program group earned an 
average of only about $0.22 more per hour than workers in the control group ($7.55, compared 
to $7.33), and only a slightly higher percentage of program group workers had health insurance 
benefits available on the job (57 percent program, compared to 54 percent control). In addition, 
job tenure was less than one year for both research groups, and the distribution of job 
occupations was similar for the two groups. 

9Q 

In quarter 16, about 22 percent worked in service occupations (such as food and health 
service); 20 percent worked in construction occupations; 11 percent worked in sales; 
13.5 percent worked as mechanics, repairers, or machinists; 12.5 percent were in clerical 
occupations; and less than 8 percent were in private household occupations (such as building and 
apartment maintenance, baby-sitting, and child care) or agricultural or forestry trades. 
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Thus, it appears that the modest program-control group differences in labor market activities 
that occurred during the initial postprogram period did not put the program group on a different 
earnings trajectory than the control group. Instead, the earnings differences faded as both groups 
gained job-specific skills through increased work experience. The earnings of both groups 
increased over time, but youths in both groups continued to have low-paying, intermittent jobs, 
as demonstrated by the very low earnings levels in the administrative records data (about $9,000 
per worker in 2000 and $9,600 per worker in 2001, according to the SER data). 

Another potential explanation for the diminishing impacts is that most program group 
members returned to their home communities after leaving Job Corps. At the 48-month follow- 
up interview point, about 73 percent of program group members (and 75 percent of control group 
members) lived less than 10 miles from where they lived at application to Job Corps. Thus, Job 
Corps does not have a large effect on student mobility to areas that offer opportunities different 
from those in areas where students come from. Consequently, the earnings impacts may have 
decreased because program and control group members tended to face the same labor market 
conditions and opportunities. 

The control group also may have caught up because more program than control group 
members were enrolled in formal education and training programs in 1999 to 2001, due to 
additional schooling sought by former Job Corps participants. In this case, the earnings impacts 
would be small temporarily because students typically have low earnings. Data are not available 
to test this explanation. However, we do not believe that the earnings impacts were greatly 
affected by differences across the research groups in school enrollment rates, because only about 
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Job Corps helps place terminating students in additional education and training programs. 
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13 percent of both the program and control groups were enrolled in school at 48 months after 
random assignment (and the school enrollment rates were similar by age group). 

Note that the impacts in 1999 to 2001 may be somewhat reduced because some control 
group members enrolled in Job Corps. Control group members were not allowed to enroll in Job 
Corps for three years after random assignment, but they could enroll afterwards. Because of 
administrative error, about 1.4 percent of all control group members enrolled in Job Corps before 
their three-year restriction period. In addition, about 4.2 percent of control group members 
enrolled in Job Corps afterward (5.3 percent for those ages 16 and 17 at application to Job Corps, 
3.2 percent for those ages 18 and 19, and 1.5 percent for those ages 20 to 24). To the extent that 
these control group members benefited from Job Corps, the impact estimates pertaining to 1999, 
2000, and 2001 may be slightly biased downward. 

As discussed in Schochet et al. (2001), it is possible to adjust for these control group 
“crossovers” by dividing the impacts per eligible applicant by the difference between the Job 
Corps enrollment rate for the program group and the crossover rate for the control group. These 
impacts pertain to eligible applicants who would enroll in Job Corps if they were assigned to the 
program group but who would not enroll if they were assigned to the control group. However, 
because the crossover rate is small and the earning impacts in 1999, 2000, and 2001 are very 
small, the adjustment procedure has little effect on the estimates. It does not change the overall 
conclusion that the impacts in these years are small and not statistically significant. 

Importantly, we believe that it is too early to conclude that the zero earnings impacts will 
persist. Little research exists on the long-term earnings growth of youths similar to those served 
by Job Corps and on the long-term earnings impacts of youth training programs. Consequently, 
we cannot discount the possibility that positive earnings impacts might re-emerge. As discussed 
below, there is some evidence that beneficial earnings impacts persisted for those ages 20 to 24 
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at application to Job Corps, but not for those younger. Thus, participation in Job Corps may 
increase the long-term earnings of the younger students (who were only in their early to mid- 20s 
in 2001) as they mature, find stable jobs, and experience the full benefits of program 
participation, both from increased vocational and academic skills gained in the program and from 
improved social skills and attitudes towards work. Furthermore, the literature on the economic 
returns to schooling, which formed the basis for our extrapolation assumption used in the 
benefit-cost analysis, typically focuses on earnings gains from additional schooling over a 
relatively long period, rather than over a very short one. This literature also focuses on 
nationally representative samples of the U.S. population and on traditional schooling. Thus, it is 
too early to conclude that this literature is not relevant to the type of training Job Corps offers 
and to the population Job Corps serves. 

It is also possible that positive earnings impacts may re-emerge as economic conditions 
change. The National Job Corps Study was conducted during a period of strong economic 
growth, with low unemployment and inflation. Between 1995 and 2001, the unemployment rate 
for the civilian population of those age 16 and older ranged from 4 to 5.5 percent. The literature 
contains some evidence that those with lower skills benefit more from a strong economy (that is, 
a tight labor market) than those with higher skills (Hoynes 1999; and Katz and Krueger 1999). 
Thus, although both program and control group members earned low wages, the strong economy 
may have favored the lower-skilled control group. Earnings impacts, however, remained small 
in 2001 as the economy started to weaken and the employment rate decreased for both the 
program and control groups (Table III.l). Thus, the early evidence suggests that the long-term 
earnings impacts probably will remain small. However, additional follow-up data need to be 
collected before definitive conclusions can be reached. 
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At this juncture, however, we need to reassess the key assumption used in the benefit-cost 
analysis that earnings gains observed during the four-year survey period will persist without 
decline. In Chapter V, we recalculate program benefits based on assumptions about the decay in 
future earnings impacts that incorporate the impact findings using the administrative records 
data. 

Finally, it is impossible to say whether estimated impacts based on survey data would have 
also disappeared in 1999, 2000, and 2001. Earnings levels are substantially higher according to 
the survey than administrative records data. Thus, if additional follow-up survey data had been 
collected, it is possible that the survey-based impact estimates may not have decayed as rapidly 
in 2000 and 2001 as those based on the administrative records data. We believe this is unlikely, 
however, for several reasons. First, the pattern of earnings impacts using the SER and survey 
data are similar during the period covered by the survey, which suggests that impact estimates 
using the survey data would have also decreased. Second, it would be difficult to interpret a 
finding of positive impacts according to the survey data, but not according to the administrative 
records data, because that would suggest that Job Corps participation has an effect on earnings 
from informal jobs that are not covered by the administrative records data, but no effect on 
earnings from formal jobs. 

Clearly, we will never know the extent to which the survey-based impacts would have 
persisted. However, understanding the sources of differences between reported earnings levels 
in the administrative and survey data can help address this issue. We address this topic in 
Chapter IV. 

B. IMPACT FINDINGS FOR SUBGROUPS 

A key finding in the main impact report is that beneficial survey-based earnings impacts in 
years 3 and 4 after random assignment were found broadly across most groups of students and 
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types of settings. Earnings gains very similar for males and females. Positive impacts were found 
for groups at special risk of poor outcomes (such as very young students, females with children, 
and older youths who did not have a high school credential at baseline). They were also found 
for groups at lower risk, such as older participants with a high school credential at baseline. 
Earnings gains were found for whites and African Americans, for residents and nonresidents, and 
for most groups defined by center characteristics (such as center size, type of operator, and 
performance level). No impacts, however, were found for Hispanic youths and 18- and 19-year- 
olds. 

As discussed, no beneficial earnings impacts based on the SER data were found in 1999 to 
2001 for the full sample. However, it is possible that the beneficial impacts found during the 
four- year survey period persisted for some population subgroups. 

Table III.5 summarizes the impact findings for key subgroups of students using the survey 
and SER data, and Table III. 6 presents results for subgroups defined by key center 
characteristics. Eor each subgroup, the tables present the ratio of average survey-to-SER earnings 
in calendar year 1998, SER-based estimated earnings impacts in calendar years 1998 to 2001, 
and survey-based estimated earnings impacts in calendar year 1998 and in year 4 after random 
assignment. Tables B.3 to B.16 display more complete subgroup results for the primary 
subgroups defined by gender, age, and residential status. 

As we discuss next, the impact findings for subgroups are similar to those for the full 
sample, with the exception of the 20- to 24-year-olds. 

1. Impact Findings for Subgroups During the Period Covered by the Survey 

In general, the pattern of the estimated subgroup impacts using the survey and SER data are 
similar in periods covered by both data sources. However, as with the full sample, the survey- 
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RATIO OF SURVEY-TO-SER MEAN EARNINGS AND EARNINGS IMPACTS 
ACCORDING TO THE SER AND SURVEY DATA, BY SUBGROUP 
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Estimated Impacts on Earnings (in 1995 Dollars)^ 

Ratio of Survey-to-SER Mean Earnings 

in 1998 Annual Social Security Earnings Records Survey Data 
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RATIO OF SURVEY-TO-SER MEAN EARNINGS AND EARNINGS IMPACTS ACCORDING 
TO THE SER AND SURVEY DATA, EOR CENTER SUBGROUPS 
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Performance Level 

High 1.65 1.64 554.8 71.5 -212.4 -273.1 929.7 1,087.8* 

Medium 1.75 1.68 480.1*** 163.3 158.1 245.3 1,218.7*** 880.5*** 

Low 1.92 1.85 76.7 496.5 563.6 605.3 491.1 669.8 




Sources: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; and (2) annual social security earnings 

records. 
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based impact estimates are larger and more often statistically significant, which results in several 
notable differences between the subgroup findings using the two data sources. 

In particular, the impact estimates in 1998 for those ages 16 and 17 at program application 
and for females are statistically significant according to the survey data but not according to the 
SER data. A possible explanation for the 16- and 17-year-olds is that these youths were more 
likely than those older to have earnings from jobs not covered in the SER data. This hypothesis is 
supported by the fact that the survey-to-SER earnings ratios are larger for the 16- and 17-year- 
olds than for older sample members, perhaps because these youths were more likely to hold 
informal (cash-only) jobs (see Chapter IV). The findings for females are puzzling, because the 
survey-to-SER earnings ratios are smaller for females than males. 

Consistent with the full-sample findings, the subgroup impact estimates measured in 
nominal terms are also larger using the survey than SER data, because average reported earnings 
levels are larger in the survey data for every subgroup. Interestingly, the survey-to-SER earnings 
ratios are largest for the 16- and 17-year-olds (as already noted) and for those who were arrested 
prior to random assignment. The ratios are also larger for males than females and for whites and 
African Americans than Hispanics. 

We also find that the survey-to-SER earnings ratios are larger for the program group than 
the control group for all subgroups except one. Consequently, the survey-based impact estimates 
in percentage terms are also consistently larger than the SER-based ones during comparable 
periods (not shown). 

2. Impact Findings for Subgroups After the Period Covered by the Survey 

We do not find any beneficial SER-based earnings impacts in 2000 and 2001 that are 
statistically significant for any subgroup. There is some evidence, however, of positive earnings 
gains for those ages 20 to 24 and those with a high school credential at program application. 
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The findings for 20- to 24-year-olds are consistent with previous project findings. Older 
students remain in Job Corps longer than younger ones. Furthermore, our interviews with 
program staff indicate that the 20- to 24-year-olds are more highly motivated and well-behaved 
than those younger. Moreover, younger students exhibit several baseline characteristics that 
suggest they are more disadvantaged and harder to serve than older students. In the baseline 
interview, a higher proportion of younger students reported having used drugs, having ever been 
arrested, living in single-parent households, and coming from families that receive public 
assistance. The younger students also enter the program with lower education levels (about 50 
percent of those age 20 to 24 have a high school credential, compared to only 13 percent of those 
younger). Moreover, the estimated impacts on total hours spent in education and training during 
the four- year survey period are larger for the older students than the younger ones, because older 
students stayed longer in Job Corps, and because a large percentage (nearly half) of the younger 
control group members attended high school after being rejected for Job Corps (Schochet et al. 
2001). Finally, the older Job Corps students typically received more hours of vocational training 
while in the program than the younger ones, which may have led to more beneficial employment 
outcomes for the older students. 

Thus, it is not surprising that the earnings gains were larger and more persistent for the 20- 
to 24-year-olds than for those younger (Tables B.3 to B.5). Positive earnings gains were found 
for the 16- and 17-year-olds soon after they left the program, but these initial gains soon 
disappeared. No earnings gains were ever found for those ages 18 and 19. Thus, Job Corps 
seems to be most effective for the oldest group of students. 

The positive findings for those with a high school credential at baseline are consistent with 
the findings for the 20- to 24-year-olds, because of the considerable overlap between youths in 



71 




these two subgroups. About 60 percent of those who completed high school or had earned a 
GED at baseline were ages 20 to 24 at baseline. 

We also find positive impact findings for the small sample of male nonresidents 
(Tables III. 5 and B.14). However, we believe that these findings are anomalous, for several 
reasons. First, the mean SER earnings of the program group were significantly higher than those 
of the control group in the period prior to random assignment (that is, in 1993 and 1994; Table 
B.14). If we net out this preexisting difference, the impact estimates for the nonresidential males 
become smaller and statistically insignificant. Second, previous studies (for example, the JTPA 
and JOBSTART evaluations) have found that disadvantaged male youths do not experience 
earnings gains from participation in education and job-training programs that offer services in a 
nonresidential setting. Thus, we believe that our findings for male nonresidents are due to small 
sample sizes. 

We find a similar pattern of findings for subgroups defined by center characteristics as for 

o 1 

the youth subgroups (Table III.6). While there is some evidence of positive earnings gains for 
those in medium-sized centers, in regions 2, 4, 6, and 9, and in low-performing centers, the 
earnings gains across subgroups generally decreased from 1998 to 2001. 

In sum, the results suggest, that based on the administrative records data, the beneficial 
subgroup impacts found in 1997 and 1998 did not persist in 1999, 2000, and 2001 for most 
subgroups, with the exception of the 20- to 24-year-olds. However, longer-term administrative 
data will need to be collected before firm conclusions can be reached about the long-term effects 

We obtained impact estimates for the center subgroups by assigning equal weight to each 
center (that is, the center rather than the eligible applicant was the unit of analysis). Thus, the 
full sample results displayed in Table III.6 differ from the full sample results displayed in 
previous tables. 
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of Job Corps on key subgroups. It still remains to be determined whether the positive earnings 
gains for the 20- to 24-year-olds will persist and whether earnings gains for the younger sample 
members will emerge as they reach their later 20s and beyond. 
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IV. RECONCILING DIFFERENCES BETWEEN FINDINGS USING THE SURVEY 
DATA AND ADMINISTRATIVE RECORDS DATA 



As discussed in the last chapter, the pattern of impact findings using the administrative and 
survey data are similar in periods covered by both data sources. However, the estimated impacts 
are larger using the survey data. To what broad factors can these differences be attributed? 

First, the differences can be attributed to reporting differences between the two data sources. 
Using comparable samples, average reported earnings levels in the survey data are nearly double 
the levels reported in the SER and UI data. Thus, the impacts — in nominal terms — using the 
survey data are much larger than those using the administrative records data. In addition, the 
reporting differences are, on average, slightly larger for the program than control group, which 
contributes to the larger impacts in percentage terms using the survey data. 

Second, the differences in the findings using the two data sources can be attributed to 
interview nonresponse bias. The impacts using the SER data are larger for survey respondents 
(about 80 percent of the sample) than for survey nonrespondents, which results in impact 
estimates based on the survey sample that are somewhat biased upward. 

We estimate that about half the difference between the percentage earnings gains in 1998 
based on the survey and SER data is due to reporting differences and about half is due to 
interview nonresponse bias. 

This chapter examines further the sources of differences between the survey-based and 
administrative-based estimates. The purpose of this analysis is to help further understand the 
impact findings discussed in the previous chapter. The chapter contains two sections. Eirst, we 
examine sources of reporting differences between the two data sources. Second, we examine 
potential reasons for interview nonresponse bias. 
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A. EXAMINING REPORTING DIFFERENCES 



There are several possible explanations for the higher reported earnings levels in the survey 
data than in administrative records data. First, informal and some formal jobs are not covered by 
the administrative records data but may be captured in the survey data. Second, some survey 
respondents may have over-reported their earnings and employment levels due to recall error or 
other reasons. Third, some employers may have inaccurately reported (or not reported) sample 
members’ earnings to the government. Finally, the administrative records data may have missed 
earnings from sample members with SSNs (or other identifying information) that were 
incorrectly reported by employers or sample members. 

To examine these explanations fully, it would be necessary to conduct a supplemental study 
to obtain detailed job information for sample members whose reported earnings levels differ 
substantially in the survey and administrative data. These interviews would collect information 
that, for example, could be used to distinguish between earnings from formal and informal jobs 
and from regular and overtime hours. Interviews with some of the youths’ employers would also 
shed light on the earnings that employers reported to the government, as well as on potential 
discrepancies between hourly wages and hours worked from the perspective of sample members 
and their employers. Data would be collected from structured interviews and from focus groups. 

Such a study is beyond the scope of this evaluation. Instead, to examine reasons for the 
reporting differences, we use available job information from the 48-month follow-up surveys. As 
discussed in Chapter II, these survey data contain some information on jobs that sample 
members held during the follow-up period. The survey, however, was not structured to gather 
sufficiently detailed information on jobs to determine earnings accurately from jobs that were 
and were not likely to have been reported to the government. Thus, our analysis is somewhat 
limited by data constraints. However, it provides important insights into the reasons that earnings 
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levels are so much higher in the survey than administrative data. These results can help guide 
future research. 

We conducted our analyses using the administrative and survey data in periods covered by 
both data sources, and using comparable samples of youths. We compared earnings and 
employment in 1998 as measured by the SER and survey data using the sample of those who 
completed the 48-month interview. For the analysis using UI data, we compared earnings in 
quarter 16 after random assignment as measured by the UI and survey data using the sample of 
those who (1) completed the 48-month interview, (2) lived in the 22 UI states for the entire four- 
year period after random assignment, and (3) did not work elsewhere. 

This section contains three parts. First, we provide descriptive information on the 
distribution of individual differences in earnings and employment as measured by the survey and 
administrative data. This analysis builds on the aggregate-level analysis presented in the previous 
chapter. Second, we attempt to identify reasons for the earnings differences in the survey and UI 
data. In particular, we compare the characteristics of jobs reported in the survey data only to 
those reported in both the survey and UI data, to help understand why quarterly employment 
rates are about 13 percentage points higher using the survey data. We also examine the extent to 
which differences in earnings per job across the two data sources vary by key job characteristics, 
to help explain why earnings levels are much higher in the survey than administrative data, even 
for those with the same number of reported jobs in both data sources. Finally, we summarize our 
results. 

1. Distribution of Individual Differences in Reported Employment Status, Number of 

Jobs, and Earnings 

In this section, we present results examining individual differences in labor market outcomes 
as measured by the survey and administrative data. Earnings differences between the two data 
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sources can be attributed to differences in (1) reported employment rates, (2) the number of jobs 
per worker, and (3) earnings per job. Thus, we discuss each of these components in turn, where 
we address the following questions: 



• What is the overlap in employment status using the survey and administrative 
records data? Were the same people employed according to both data sources? 

• Did the number of reported jobs differ by data source? How often was the number 
of reported jobs greater in the survey than UI data? 

• What is the distribution of individual earnings differences using the survey and 
administrative records data? Are earnings differences due to a small number of 
people who reported much higher earnings in the survey, or are they more common? 
To what extent can differences in overall reported earnings levels in the survey and 
UI records data be explained by (1) higher employment levels reported in the survey, 
(2) more jobs per worker reported in the survey, and (3) higher reported earnings per 
job in the survey? 

• To what extent do answers to these questions differ for the program and control 
groups? Why are the ratios of survey-based to administrative-based earnings levels 
slightly higher for the program than control group? 



a. Differences in Reported Employment Status 

Results Using the SER Data. As discussed in Chapter III, the overall annual employment 
rate in 1998 is slightly higher according to the SER data than the survey data for both research 
groups. For the program group, the employment rate is 86 percent using the SER data and 
81 percent using the survey data, and for the control group, it is 84 percent using the SER data 
and 79 percent using the survey data (Table III.3). Consequently, the higher average earnings 
levels reported in the survey than SER data are due to higher earnings per worker rather than to 
higher employment rates in the survey data. 

Although the estimated annual employment rates in 1998 are similar using the survey and 
SER data, it may not be the case that the same youths were employed according to both data 
sources. We do find, however, considerable overlap in the youths’ employment status across the 
two data sources (Table IV. 1). In 1998, about 75 percent were employed and 10 percent were not 
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TABLE IV. 1 



OVERLAP IN EMPLOYMENT STATUS IN 1998 AS REPORTED IN THE SURVEY AND SER DATA, 

BY RESEARCH STATUS AND SUBGROUP 
(Percentages) 





Employed 


in 1998 According to Survey Data/ According to SER Data 


Subgroup 


YesA^es 


Yes/No 


No/Yes 


No/No 


Yes/Yes or 
No/No 


Pull Sample 


Program group 


76.3 


5.0 


9.7 


9.0 


85.3 


Control group 


73.9 


5.2 


10.7 


10.2 


84.1 


Males 


Program group 


77.4 


5.4 


9.4 


7.8 


85.2 


Control group 


75.5 


5.4 


10.0 


9.2 


84.7 


Eemales 


Program group 


75.1 


4.5 


10.1 


10.3 


85.4 


Control group 


71.3 


4.9 


12.0 


11.8 


83.1 


16 to 17 at Application 


Program group 


72.6 


5.7 


11.5 


10.2 


82.8 


Control group 


72.0 


5.4 


11.8 


10.8 


82.8 


18 to 19 at Application 


Program group 


76.5 


5.0 


9.2 


9.3 


85.8 


Control group 


74.4 


5.6 


9.7 


10.4 


84.8 


20 to 24 at Application 


Program group 


81.4 


4.0 


7.8 


6.8 


88.2 


Control group 


76.3 


4.4 


10.3 


9.1 


85.4 


White, Non-Hispanic 


Program group 


82.3 


4.4 


8.1 


5.2 


87.5 


Control group 


81.9 


3.5 


8.6 


6.0 


87.9 


Black, Non-Hispanic 


Program group 


73.7 


5.2 


10.6 


10.5 


84.2 


Control group 


69.0 


6.0 


12.9 


12.1 


81.1 


Hispanic 


Program group 


75.9 


4.7 


9.6 


9.9 


85.8 


Control group 


76.0 


5.3 


8.4 


10.4 


86.4 


Full Sample Size 


(Program/Control) 


6,772/4,451 


6,772/4,451 


6,772/4,451 


6,772/4,451 


6,772/4,451 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month 

interviews, and 1998 social security earnings records for those who completed 48-month interviews. 

Note: All figures are unweighted. 



79 




employed according to both data sources, resulting in an “agreement rate" of about 85 percent.^^ 
About 5 percent were employed according to the survey data only, and about 10 percent were 
employed according to the SER data only. Stated differently, about 94 percent of those who were 
employed based on the survey data were also employed according to the SER data, and about 
88 percent of those who were employed according to the SER data were also employed 
according to the survey data. 

These high agreement rates suggest that both data sources captured the broad employment 
experiences of the sample. They also reinforce our conclusion that the correct SSNs of our 
sample members were matched to the SER data system. 

The overall agreement rate is similar for the program and control groups, although it is 
slightly higher for the program group (85.3 percent program, 84.1 percent control; Table IV. 1). 
In addition, the agreement rate does not differ substantially by gender, age, and race and 
ethnicity. It is slightly higher, however, for those 20 to 24 years old at program application (who 
were 23 to 27 years old in 1998) than for those younger, and for whites than their counterparts. 
The results for the 20- to 24-year-olds are not surprising, because these people typically had 
more stable, higher-paying jobs than younger workers and thus were more likely to have 
earnings reported in the administrative records data. 

Results Using the UI Data. As discussed, the employment rate in quarter 16 after random 
assignment is substantially higher according to the survey than UI data (Table III.4). Eor the 
program group, the employment rate is about 7 1 percent using the survey data, compared to only 
58 percent using the UI data. Eor the control group, it is 69 percent using the survey data, 

T9 

The expected agreement rate would be about 73 percent if employment status in the 
survey data were independent of employment status in the SER data. 
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compared to only 55 percent using the UI data. Thus, quarterly earnings differences between the 
survey and UI data are due partly to higher quarterly employment rates using the survey data. 

It is not possible to directly compare annual employment rates using the survey and UI data, 
because the periods the two data sources cover do not overlap enough. However, there is indirect 
evidence that the annual employment rates using the two data sources are similar. First, the 
annual employment rate in 1998 using the survey data is similar to the annual employment rate 
in 1999 using the UI data (Table III.l). Second, the difference between the combined six-month 
employment rate in quarters 15 and 16 (the maximum period of overlap) using the survey and UI 
data is much smaller than the difference between the two employment rates in quarter 16 only, 
which suggests that the difference in annual employment rates would be even smaller. 

Our finding that quarterly employment rates are higher using the survey than using 
administrative data, but that annual employment rates are similar, suggests that the 
administrative data did not capture earnings from short-term informal (second) jobs held by the 
sample. These findings also suggest that sample members reported during the survey that they 
were employed in jobs longer than they actually were (perhaps because of recall error). The 
reporting differences may also be due in part to errors in matching our sample members to the 
state’s UI data systems because of inaccurate SSNs. 

The agreement rate in employment status in quarter 16 according to the survey and UI data 
is about 70 percent (Table IV. 2). According to both data sources, about 50 percent of sample 
members were employed and 20 percent were not. This figure is lower than the 85 percent 
agreement rate for annual earnings using the SER data. However, about 85 percent of workers in 

If employment status in the survey data were independent of employment status in the UI 
data, the expected agreement rate would be about 53 percent. 
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TABLE IV.2 



OVERLAP IN EMPLOYMENT STATUS IN QUARTER 16 AETER RANDOM ASSIGNMENT AS 
REPORTED IN THE SURVEY AND UI DATA, BY RESEARCH STATUS AND SUBGROUP 

(Percentages) 





Employed 


in Quarter 16 According to Survey Data / According to UI Data 


Subgroup 


YesA^es 


Yes/No 


No/Yes 


No/No 


Yes/Yes or 
No/No 


Pull Sample 


Program group 


50.0 


21.1 


8.8 


20.1 


70.1 


Control group 


47.1 


22.0 


9.0 


21.9 


69.0 


Males 


Program group 


47.4 


23.8 


9.2 


19.6 


67.0 


Control group 


45.8 


24.5 


9.4 


20.3 


66.1 


Pemales 


Program group 


52.9 


18.0 


8.4 


20.7 


73.6 


Control group 


49.1 


18.4 


8.3 


24.2 


73.3 


16 to 17 at Application 


Program group 


45.0 


21.1 


9.9 


23.9 


69.0 


Control group 


44.5 


23.4 


8.6 


23.5 


68.0 


18 to 19 at Application 


Program group 


51.9 


20.4 


8.2 


19.5 


71.3 


Control group 


49.6 


20.7 


8.7 


21.0 


70.5 


20 to 24 at Application 


Program group 


54.5 


21.7 


8.0 


15.8 


70.3 


Control group 


48.4 


21.4 


9.7 


20.4 


68.8 


White, Non-Hispanic 


Program group 


53.5 


23.8 


8.4 


14.3 


67.9 


Control group 


53.6 


22.3 


8.0 


16.1 


69.7 


Black, Non-Hispanic 


Program group 


50.7 


17.0 


9.7 


22.6 


73.3 


Control group 


45.0 


18.3 


11.4 


25.3 


70.4 


Hispanic 


Program group 


41.4 


30.2 


7.0 


21.4 


62.8 


Control group 


45.7 


30.9 


3.9 


19.6 


65.3 


Full Sample Size 


(Program/Control) 


2,256/1,508 


2,256/1,508 


2,256/1,508 


2,256/1,508 


2,256/1,508 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 
48-month interview; (2) signed the records release consent form, (3) lived in the 22 states selected for 
UI data collection for the entire 48 months after random assignment, and (4) did not work outside the 
22 states. 

Note: All figures are unweighted. 
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the UI wage records were also employed according to the survey data. Thus, there is 
considerable overlap in employment status using the survey and UI data among those identified 
as workers in the UI wage records. 

Finally, the quarter 16 agreement rate is similar for the program and control groups 
(Table IV. 2). Furthermore, the agreement rate does not differ substantially across subgroups, 
although it is slightly lower for males than females and for Hispanics than whites and African 
Americans. 

Summary. In sum, we find using the annual SER data that most sample members who were 
workers according to survey data were also workers according to the SER data. There is less 
overlap in employment status in quarter 16 according to the survey and UI data, because the 
quarterly employment rate is much higher using the survey than UI data. However, most workers 
in the UI data are also considered to be workers in the survey data. Einally, the overlap in 
employment status is very similar for program and control group members and for key 
subgroups. There is some evidence, however, that agreement rates are slightly higher for older 
sample members than younger ones (according to the SER data) and for females than males 
(according to the UI data). 

b. Differences in the Reported Number of Jobs 

The higher reported earnings levels in the survey than UI data may be due not only to 
differences in reported employment rates, but also to differences in the reported number of jobs 
held by workers. The UI wage records contain earnings for each reported job held by sample 
members. Thus, we compared the number of jobs held by sample members in quarter 16 using 
the survey and UI data. The sample for this analysis included those who were employed 
according to both data sources. 
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Substantially more jobs are reported in the survey data than in the UI data (Table IV. 3). The 
average number of jobs held per worker is 1.2 jobs according to the survey data, compared to 1.0 
job according to the UI data. Similarly, for about 20 percent of workers, the number of jobs 
reported in the survey is greater than the number reported in the UI wage records. More jobs are 
reported in the UI data for only about 2 percent of workers. These figures are very similar by 
research status and subgroup. 

These findings suggest that a nontrivial percentage of sample members had earnings from 
second jobs that were not reported to the state UI agencies, and that these jobs accounted for 
some of the differences in earnings levels as measured by the survey and UI data. 

c. Differences in Reported Earnings 

Table IV. 4 displays summary statistics for differences between individual earnings as 
reported in the survey and administrative data. We display mean differences, the standard 
deviation of the differences, and percentiles of the distribution of differences. We also display 
mean earnings levels as measured by the administrative data, so that the earnings differences can 
be put into percentage terms. The figures are presented for the full sample, for those employed 
according to both data sources, and for workers with the same number of reported jobs. The 
statistics are presented separately for the program and control groups. Because the goal of this 
descriptive analysis is to examine reporting differences at the individual level, sample weights 
were not used in the analysis. 

Earnings levels are substantially higher using the survey data than the administrative records 
data for most sample members (Table IV. 4). For the full sample, mean annual earnings in 1998 
are about $3,500 to $4,000 higher according to the survey data than the SER data, and the 
differences are about $4,700 to $5,000 using the sample of those who were employed according 
to both data sources. Similarly, mean earnings in quarter 16 are about $1,200 higher in the 
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TABLE IV.3 



NUMBER OE JOBS WORKED IN QUARTER 16 ACCORDING TO THE SURVEY AND UI DATA, AMONG 
THOSE EMPLOYED ACCORDING TO BOTH DATA SOURCES, 

BY RESEARCH STATUS AND SUBGROUP 





Average Number of Jobs 
Worked 


Difference Between the Number of Jobs Reported in the 
Two Data Sources (Percentages) 


Subgroup 


Survey Data 


UI Data 


Same Number in 
Both Data Sources 


More in Survey 
Data 


More in UI 
Data 


Pull Sample 


Program group 


1.2 


1.0 


78.9 


19.7 


1.4 


Control group 


1.2 


1.0 


76.9 


21.1 


1.9 


Males 


Program group 


1.2 


1.0 


79.5 


18.8 


1.7 


Control group 


1.3 


1.0 


75.0 


22.9 


2.1 


Eemales 


Program group 


1.2 


1.0 


78.1 


20.8 


1.1 


Control group 


1.2 


1.0 


79.7 


18.6 


1.7 


16 to 17 at Application 


Program group 


1.2 


1.0 


78.7 


19.7 


1.6 


Control group 


1.3 


1.0 


76.7 


22.9 


0.4 


18 to 19 at Application 


Program group 


1.2 


1.0 


81.6 


17.0 


1.4 


Control group 


1.2 


1.0 


76.5 


20.0 


3.5 


20 to 24 at Application 


Program group 


1.3 


1.0 


76.3 


22.6 


1.1 


Control group 


1.2 


1.0 


77.8 


20.0 


2.2 


White, Non-Hispanic 


Program group 


1.3 


1.0 


74.2 


24.2 


1.7 


Control group 


1.3 


1.0 


74.1 


25.9 


0.0 


Black, Non-Hispanic 


Program group 


1.2 


1.0 


78.4 


19.9 


1.7 


Control group 


1.3 


1.0 


74.6 


22.7 


2.7 


Hispanic 


Program group 


1.2 


1.0 


84.3 


15.1 


0.5 


Control group 


1.1 


1.0 


83.4 


13.8 


2.8 


Full Sample Size 


(Program/Control) 


1,128/711 


1,128/711 


1,128/711 


1,128/711 


1,128/711 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 1999 and 

2000 from 22 randomly selected states. The sample includes those who (1) were employed according to both 
data sources; (2) signed the records release consent form, (3) lived in the 22 states selected for UI data 
collection for the entire 48 months after random assignment, and (4) did not work outside the 22 states. 

Note: All figures are unweighted. 
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TABLE IV .4 



SUMMARY STATISTICS FOR DIFFERENCES BETWEEN EARNINGS AS REPORTED IN THE SURVEY AND 

ADMINISTRATIVE DATA, BY RESEARCH STATUS 





Difference Between Survey and SER 
Earnings in 1998 
(in 1995 Dollars) 


Difference Between Survey and UI 
Earnings in Quarter 16 After Random 
Assignment 
(in 1995 Dollars) 


Summary Statistic (Unweighted) 


Program Group 


Control Group 


Program Group 


Control Group 


Full Sample 


(Mean SER or UI Earnings Level) 


(5,825.1) 


(5,636.7) 


(1,432.2) 


(1,417.2) 


Mean Earnings Difference 


3,983.8 


3,501.3 


1,287.1 


1,139.8 


Standard Deviation of Difference 


8,111.6 


7,670.4 


2,606.1 


2,645.2 


Percentile of the Distribution of 
Differences 


10 th 


-2,149.9 


-2,246.9 


-647.7 


-694.9 


25th 


0.0 


-45.5 


0.0 


0.0 


50th 


1,859.6 


1,559.0 


468.6 


368.4 


75th 


6,390.8 


5,827.0 


2,376.7 


2,241.5 


90th 


13,437.8 


12,438.3 


4,414.9 


4,307.4 


Sample Who Were Employed 
According to Both Data Sources 


(Mean SER or UI Earnings Level) 


(7,428.8) 


(7,155.9) 


(2,640.1) 


(2,709.6) 


Mean Earnings Difference 


5,057.8 


4,682.9 


1,268.5 


1,062.2 


Standard Deviation of Difference 


7,848.7 


7,944.2 


2,464.3 


2,214.3 


Percentile of the Distribution of 
Differences 


10 th 


-1,411.4 


-1,713.2 


-610.8 


-747.8 


25th 


602.3 


466.3 


42.2 


33.8 


50th 


3,247.2 


2,938.1 


850.4 


794.5 


75th 


7,733.8 


7,397.6 


2,105.9 


2,000.7 


90th 


14,193.1 


13,761.4 


3,868.3 


3,334.6 


Sample Who Were Employed and 
Had the Same Number of Jobs 
According to Both Data Sources 


(Mean SER or UI Earnings Level) 


NA 


NA 


(2,646.3) 


(2,709.3) 


Mean Earnings Difference 


NA 


NA 


1,084.4 


926.3 


Standard Deviation of Difference 


NA 


NA 


2,340.8 


1,859.9 


Percentile of the Distribution of 
Differences 


10 th 


NA 


NA 


-644.5 


-747.8 


25th 


NA 


NA 


-4.2 


-27.9 


50th 


NA 


NA 


684.8 


564.1 


75th 


NA 


NA 


1,779.0 


1,658.9 


90th 


NA 


NA 


3,350.9 


2,917.7 


Sample Size 


6,772 


4,451 


2,645 


1,737 



Source: (I) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month interviews; (2) 

1998 social security earnings records for those who completed 48-month interviews; and (3) quarterly UI earnings 
records in 1999 and 2000 from 22 randomly selected states using those who signed the records release consent form, 
lived in the 22 states for the entire 48 months after random assignment, and did not work outside the 22 states. 

Notes: All figures are unweighted. 

NA = Not applicable because the SER data do not contain earnings from individual jobs, but only combined earnings from all jobs. 
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survey than UI data for each of the analysis samples. Median differences are smaller than 
mean differences, because reporting differences are large and positive for a substantial fraction 
of sample members (that is, the distribution of differences is skewed to the right). 

Most important, we find that survey-based earnings per job are larger than administrative- 
based earnings for about 75 percent of workers. This result is similar for the program and control 
groups. Thus, differences in reported earnings are common. They are not due to a small number 
of people who reported much higher earnings in the survey. 

d. Decomposition of UI and Survey Earnings Differences into Their Component Parts 

Differences in quarter 16 earnings levels based on the survey and UI data can be 
decomposed into differences due to (1) earnings per job, (2) the number of jobs per worker, and 
(3) employment levels. To calculate the relative contribution of each of these three components, 
we express the overall mean difference in survey-based and UTbased earnings as follows: 

(3) {Es-~Eu)=~—^Ws-^^Wu, 

Js ITs Ju Wu 

where Es is mean earnings using the survey data, Eu is mean earnings using the UI data, the 
Ji (i = S,U) variables represent the mean number of reported jobs from the two data sources, and 
the Wi variables represent employment rates according to the survey and UI data. After adding 
and subtracting relevant terms, the overall mean earnings difference can be expressed as a 
weighted sum of the ratios in the right-hand side of equation (3) as follows: 

The full sample figures are similar to the figures for the sample of workers only. This is 
because in the calculation of the full sample figures, the positive earnings differences for the 20 
percent of the sample who are workers according to the survey but not the UI data are offset by 
the zero differences for the 20 percent who are not workers according to both data sources. 
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(4) {Es-~Eu) = {—-—)p, + {—-—)p, + {Ws-Wv)p„ 
Js Ju Ws Wu 



where, 



J S , 7 , Eu -j Eu J S 35 

(5) Pi = Tys, P 2 = Wu, P3= . 

Ju Ju Ws 



Equations (4) and (5) can then be used to decompose overall mean earnings differences into its 
component parts. 

We find that all three components play an important role in explaining the overall mean 
earnings difference as measured by the survey and UI data, and this important result holds for 
both the program and control groups (Table IV. 5). For the program group, about 45 percent of 
the overall mean earnings difference is due to differences in mean earnings per job, 25 percent is 
due to differences in the mean number of jobs per worker, and 30 percent is due to differences in 
employment rates. The corresponding figures for the control group are 39, 24, and 38 percent, 
respectively. Thus, the earnings per job component is a somewhat more important factor for the 
program than control group, whereas the employment rate component is a somewhat more 
important factor for the control group. 

All three factors also play an important role in explaining the overall earnings differences 
across key subgroups, although there are some notable differences by age and race and ethnicity 
(Table IV. 5). The earnings per job component is less important for the older sample members 

Equation (5) displays one of six possible combinations of pj, p 2 , and ps that satisfy 
equation (4). Our decomposition results discussed below are based on the average of the results 
based on each of the six combinations. (Each combination, however, produced very similar 
results.) 
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TABLE IV.5 



DECOMPOSITION OE THE DIEEERENCE IN MEAN EARNINGS IN QUARTER 16 BASED ON 
THE SURVEY AND UI DATA, BY RESEARCH STATUS AND SUBGROUP 



Percentage of the Difference in Mean Earnings in Quarter 16 Based on the 
Survey and UI Data That Is Due to Differences in: 





Employment 


Mean Number of Jobs 


Mean Earnings 


Subgroup 


Rates 


Per Worker 


Per Job 



Eull Sample 



Program group 


30 


25 


45 


Control group 


38 


24 


39 


Males 


Program group 


32 


21 


47 


Control group 


37 


21 


42 


Eemales 


Program group 


28 


33 


39 


Control group 


43 


31 


27 


16 to 17 at Application 


Program group 


26 


21 


53 


Control group 


35 


21 


44 


18 to 19 at Application 


Program group 


31 


21 


48 


Control group 


32 


23 


44 


20 to 24 at Application 


Program group 


36 


33 


31 


Control group 


49 


26 


25 


White, Non-Hispanic 


Program group 


32 


22 


46 


Control group 


33 


31 


36 


Black, Non-Hispanic 


Program group 


19 


27 


54 


Control group 


27 


29 


44 


Hispanic 


Program group 


61 


21 


17 


Control group 


62 


9 


28 



Full Sample Size 

(Program/Control) 2,754/1,872 2,754/1,872 2,754/1,872 



Source: (1) Baseline and 12-, 30-, and 48-month follow-up interview data for those who completed 48-month 

interviews; (2) 1998 social security earnings records for those who completed 48-month interviews; 
and (3) quarterly UI earnings records in 1999 and 2000 from 22 randomly selected states using those 
who signed the records release consent form, lived in the 22 states for the entire 48 months after 
random assignment, and did not work outside the 22 states. 

Notes: All figures were calculated using sample weights to adjust for the sample and survey designs. 
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than the younger ones, perhaps because the older sample members were more likely to work jobs 
with stable hours. In addition, the earnings per job component is much less important for 
Hispanics than the employment rate component, and the reverse holds for African Americans. 

2. Explanations for Higher Employment Rates in the Survey Data than in the UI Data 

We have shown that estimated employment levels in quarter 16 are about 13 percentage 
points higher according to the survey than UI data for both the program and control groups. 
What caused these reporting differences? 

We have already discussed two partial explanations. First, the Ul-based employment rates 
could be lower because of financial incentives for employers to underreport earnings to state UI 
programs. Second, the Ul-based rates could be lower because of errors in matching our sample 
members to the states’ UI data systems. These explanations are supported somewhat by the 
figures in Table III.l, which show that annual employment rates in 1999, 2000, and 2001 are 
about 5 percentage points higher according to the SER data than the UI data, even though job 
coverage is similar for the OASDI and UI programs. These explanations, however, account for 
only a portion of the difference between the quarter 16 employment rate as measured by the 
survey and UI data. 

We offer two other possible explanations for the lower Ul-based employment rate. First, 
^ome, formal jobs are not covered under the UI program. Hence, the earnings of sample members 
who held such jobs may not have been reported to the UI agencies but may have been reported in 
the survey. Second, informal jobs are not covered under the UI program, although earnings from 
these casual or cash-only jobs may have been reported in the survey. 

Next, we examine these two hypotheses in more detail. Specifically, we address the 
following questions: 
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• To what extent are higher employment rates in the survey data due to formal jobs 
that are not covered by the UI program? Does the overlap in employment status 
using the survey and UI data (that is, agreement rates) differ by occupation? 

• To what extent are higher employment rates in the survey data due to informal 
jobs? What are the characteristics of jobs (such as job tenure, hourly wages, weekly 
earnings, usual hours worked, available fringe benefits) reported in the survey data 
but not in the UI data? What are the demographic characteristics and employment 
experiences of those who held these informal jobs? 

• To what extent do the answers to these questions differ for the program and control 
groups? Is there any evidence that the program group was more likely than the 
control group to have held informal jobs and formal jobs not covered by the UI 
program? 



a. Examining Formal Jobs Not Covered in the UI Wage Records 

The UI wage records do not cover workers in some formal jobs. These workers include 
federal workers, military staff, self-employed people, agricultural labor (except workers on large 
farms), and domestic service workers (whose employers paid less than $1,000 in wages in any 
calendar quarter). 

About 17 percent of workers in both the program and control groups reported in the survey 
that they worked in these “uncovered” sectors during their most recent jobs in quarter 16. These 
workers were identified using survey information on reported job occupations (which were open- 
ended responses coded into three-digit SOC codes) and on reported types of employers. About 
2 percent of workers in the survey sample reported working for the federal government, about 1 
percent were in the military, 5 percent were self-employed, 2 percent worked in agricultural 
occupations, 7 percent worked in private household occupations, and 1 percent worked without 



Federal workers and military staff are eligible to receive UI benefits. Their earnings are 
not reported to state UI agencies, however, and so are not in the UI wage records. 

-57 

The type of employer data provided information on whether the respondent was working 
for a private company; on active military duty; a federal, state, or local government employee; 
self-employed; working without pay or in a family business or as a favor, or other. 
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-no 

pay (see columns 2 and 4 in Table IV. 5). The presence of these “uncovered” workers would 
completely explain the gap between the survey-based and Ul-based employment rates if earnings 
for these workers were not reported in the UI data (and if earnings for all other workers were 
reported in the UI data). 

Before proceeding, we note that, based on national data, we expect that some sample 
members in these “uncovered” jobs were actually covered by the UI program. UI wage records 
cover about 94 percent of workers nationally (U.S. General Accounting Office 2002), but U.S. 
workers in the “uncovered” sectors described above comprise more than 6 percent of all U.S. 
workers. Thus, some of these “uncovered” U.S. workers must have actually been covered by 
the UI program. For example, some farmers and domestic workers are covered by the UI 
program, although it is not possible to determine from published statistics (or our survey data) 
the number of such workers. Furthermore, there is often ambiguity about reported self- 
employment status. 

To examine the extent to which uncovered formal jobs explain the lower UTbased than 
survey-based employment rate, we calculated the percentage of workers according to the survey 
data who were also workers according to the UI data (that is, agreement rates for survey-based 
workers) by occupation, type of employer, and research status. Table IV. 6 presents these figures 
for the full sample, and Table C.l presents them for subgroups. 

The sum of the percentages in each type of uncovered job (19.3 percent) is greater than 
the 17 percent figure who worked in any of them, because some youths who worked in private 
household and agriculture occupations reported that they were self-employed. 

TQ 

In 1999, 2.1 percent of all workers nationally reported working for the federal 
government, 7 percent were self-employed, 3.5 percent worked in agricultural-related 
occupations, and 1 percent worked in private household occupations (Statistical Abstract of the 
United States 2000). 
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TABLE IV.6 



AGREEMENT RATES BETWEEN THE SURVEY AND UI DATA BY OCCUPATION AND TYPE OE 
EMPLOYER IN THE MOST RECENT JOB HELD IN QUARTER 16, BY RESEARCH STATUS 





Program Group Members Employed 
According to the Survey Data 


Control Group Members Employed 
According to the Survey Data 


Job Characteristic According 
to Survey 


Percentage 
with Job 
Characteristic 


Percentage 
Also Employed 
According to the 
UI Data 


Percentage 
with Job 
Characteristic 


Percentage 
Also Employed 
According to the 
UI Data 


Occupation 

Services 


22.0 


72.5 


21.0 


75.1 


Sales 


10.1 


74.6 


12.8 


67.9 


Construction 


20.3 


66.0 


19.4 


66.8 


Private household 


6.5 


59.6 


7.8 


52.8 


Clerical 


12.3 


73.6 


13.0 


73.6 


Mechanics/repairers/ 

machinists 


14.3 


68.8 


13.4 


63.7 


Agriculture 


2.2 


60.6 


2.3 


63.2 


Other 


12.3 


68.5 


10.4 


68.3 


Type of Employer 
Private company 


80.5 


73.7 


80.7 


n 

69.7 


Military 


1.4 


12.9 


0.5 


0.0 


Eederal government 


2.3 


57.4 


2.1 


71.8 


State government 


4.5 


66.7 


4.9 


70.9 


Local government 


3.7 


75.0 


4.0 


74.6 


Self-employed 


5.3 


41.8 


5.3 


32.9 


Working without pay in a 


1.0 


30.3 


1.2 


60.7 


family business or as a favor 
Other 


1.4 


35.1 


1.2 


84.0 


Sample Size 


1,870 


1,870 


1,198 


1,198 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for UI 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

Note: All figures were calculated using sample weights to adjust for the sample and survey designs. 
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The overall agreement rate for all survey-based workers is slightly less than 70 pereent for 
both the program and control groups. Stated differently, 70 percent of those classified as workers 
according to the survey data are also classified as workers according to the UI data. 

Agreement rates are somewhat lower for workers in the “uncovered” sectors than for 
“covered” workers, but the differences are smaller than expected (Table IV. 6). For example, 
among the program group, the agreement rate for survey-based workers is about 60 percent for 
those employed in private household and agriculture occupations, 60 percent for federal workers, 
42 percent for those self-employed, and only 13 percent for those in the military. Nonetheless, 
many of these “uncovered” workers had reported earnings in the UI data. Furthermore, the 
agreement rate for “covered” workers is only about 72 percent. Stated differently, about 
28 percent of “covered” workers do not have reported earnings in the UI data. The pattern of 
findings is similar for the program and control groups. Furthermore, despite differences in the 
types of jobs held by males and females and younger and older sample members, the pattern of 
estimated agreement rates is similar across subgroups (Table C.l). 

These disappointing findings could be due in part to errors in misclassifying jobs reported in 
the survey into occupational categories, due to limited verbatim survey information on the nature 
and title of jobs that sample members held. However, because the agreement rates do not differ 
substantially for the “covered” and “uncovered” sectors, it appears that the noncoverage of 
formal jobs in the UI data accounts for only a small portion of the gap between the quarter 16 
employment rate as measured by the survey and UI data. 

b. Examining Informal Jobs 

Another possible explanation for the lower quarter 16 employment rate in the UI data is that 
earnings from informal (casual or cash-only) jobs are covered in the survey data but not in the UI 
data. Under-the-table or off-the-books payments from informal jobs were likely to have been 
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reported in the survey, because survey respondents were asked to provide information on “paid 
full-time or part-time jobs that they may have had, including odd jobs, paid baby-sitting jobs, 
military service, work in their own businesses, or other types of jobs they may have had on a 
regular basis.” Some survey respondents might not have reported earnings from informal jobs 
that were performed on an irregular basis (because the survey asked for information on jobs 
performed on a regular basis). We believe this was uncommon, however, because many sample 
members reported earnings from jobs that were performed only once or twice (for example, 
baby-sitting or lawn-mowing). 

In partial support of the informal-job explanation, the agreement rates for survey workers 
were somewhat lower in occupations in which we might expect informal jobs to be more 
common (Table IV. 6). For example, the agreement rate for those in construction and mechanical- 
related trades was about 65 percent, compared to about 72 percent for those in service, sales, and 
clerical occupations. 

To examine further the extent to which informal jobs explain the lower UTbased 
employment rate, we compared the characteristics of jobs reported in both the survey and UI data 
with the characteristics of jobs reported in the survey data only^^ Job characteristics were 
obtained from the survey data, and pertain to the most recent job held in quarter 16 after random 
assignment. We expect that the survey-only jobs were more likely to have been informal jobs 
than those reported in both data sources. Thus, we anticipate that the survey-only workers had 



For this analysis, we omitted those who had Ul-based jobs but more survey-based jobs, 
because it was often difficult to accurately determine which survey-based job corresponds to 
which UTbased job and, thus, which survey jobs are the “matched” jobs and which are the 
“unmatched” ones. 
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(1) shorter job tenure, (2) fewer hours worked per week, (3) lower hourly wages, (4) lower 
weekly earnings, and (5) fewer benefits available on the job. 

These hypotheses are only weakly supported by the data (Table IV.7). The survey-only 
group was more likely to have had less than one month of job tenure, but it was also more likely 
to have at least one year of job tenure. Consequently, average job tenure was similar for the two 
groups. Similarly, the distributions of usual hours worked per week were similar for the two 
groups of workers. However, average hourly wages were somewhat lower for the survey-only 
group ($7.20, compared to $7.30 to $7.60 for those employed according to both data sources), 
and survey-only workers were much more likely to have hourly wages below $4.50. In addition, 
survey-only workers were much less likely to have health insurance, paid vacation, and 
retirement benefits available on the job. We find a similar pattern for the program and control 
groups and by age and gender (Table C.2). 

These results provide some evidence that the survey-only jobs were more likely than the 
jobs reported in both data sources to be informal jobs. However, the differences in the 
characteristics of jobs held by these two groups of workers are not as large as expected. 

c. Multivariate Analysis 

Thus far, we have examined job characteristics one at a time to assess the extent to which 
informal and uncovered formal jobs account for the much lower quarterly employment rate using 
the UI than survey data. These job characteristics, however, are correlated with each other and 
with other worker characteristics. Thus, we also estimated multivariate logistic regression 
models to examine the effects of worker and job characteristics on the probability that a job 
reported in the survey data is also reported in the UI data, controlling for the effects of other 
characteristics. The sample for this analysis included those who were employed in quarter 16 
according to the survey data. The dependent variable was set to 1 for those who also had a job 
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TABLE IV.7 



JOB CHARACTERISTICS IN QUARTER 16 EOR THOSE EMPLOYED ACCORDING TO BOTH 
THE SURVEY AND UI DATA AND ACCORDING TO THE SURVEY DATA ONLY, 

BY RESEARCH STATUS 
(Percentages) 





Program Group 


Control Group 


Job Characteristic 
According to Survey Data 


Employed in Both 
Survey and UI Data“ 


Employed in 
Survey Data Only 


Employed in Both 
Survey and UI Data“ 


Employed in 
Survey Data Only 


Number of Months on Job 


Less than 1 


4.5 


10.6 


5.2 


11.9 


1 to 3 


33.2 


30.8 


32.6 


30.4 


3 to 6 


21.8 


20.1 


22.7 


17.6 


6 to 12 


25.0 


20.0 


24.8 


21.4 


12 or more 


15.5 


18.6 


14.8 


18.7 


(Average months) 


12.5 


12.7 


12.2 


12.8 


Usual Hours Worked per 
Week 


Less than 20 


2.7 


4.3 


2.2 


6.9 


20 to 30 


5.8 


8.3 


8.5 


5.1 


30 to 39 


12.7 


14.4 


11.8 


13.5 


40 


41.9 


32.4 


40.5 


29.6 


More than 40 


36.9 


40.5 


37.1 


44.8 


(Average hours) 


42.4 


43.0 


42.4 


42.9 


Hourly Wage 


Less than $4.50 


2.9 


10.7 


3.3 


7.6 


$4.50 to $6.00 


26.2 


26.6 


30.8 


32.0 


$6.00 to $7.50 


27.7 


21.6 


30.1 


23.0 


$7.50 to $9.00 


24.3 


20.2 


19.2 


18.1 


$9.00 or more 


18.9 


21.0 


16.6 


19.3 


(Average hourly wage in 


1995 dollars) 


7.50 


7.20 


7.30 


7.20 


Average Weekly Earnings 


(in 1995 dollars) 


323.10 


310.40 


308.60 


315.10 


Benefits Available on Job 


Health insurance 


63.7 


47.4 


60.4 


46.0 


Paid vacation 


67.6 


56.3 


63.6 


60.7 


Retirement or pension 


benefits 


53.6 


40.3 


45.0 


37.5 


Sample Size 


860 


475 


524 


332 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for UI 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

Note: All figures were calculated using sample weights to adjust for the sample and survey designs. 

“Includes only those with the same number of reported jobs in both data sources. 
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reported in the UI data and zero for those who did not. We excluded from the analysis those who 
had a Ul-based job but more survey-based jobs, because it was difficult to accurately match 
specific survey-based jobs to specific Ul-based jobs. 

The regression models produced regression-adjusted agreement rates for subgroups of 
survey-based workers. The explanatory variables in the models included a large number of 
worker characteristics across which we posited agreement rates might vary, as well as job 
characteristics. Five categories of explanatory variables were included in the models: 
(1) demographic characteristics, (2) education and training experiences, (3) employment and 
earnings experiences during the four-year follow-up period, (4) welfare receipt during the 
follow-up period, and (5) arrest and drug use experiences during the follow-up period. Separate 
regression models were estimated for the program and control groups. 

Table IV. 8 displays estimated marginal probabilities and their significance levels for each 
explanatory variable (relative to the pertinent omitted variable). Thus, the marginal probabilities 
represent differences in regression-adjusted agreement rates between various groups. The overall 
agreement rate is 63 percent for the program group and 61 percent for the control group. 

The results from the multivariate models are similar to the univariate results presented above 
for both the program and control groups. Agreement rates are significantly lower for workers 
who are (1) self-employed (because these jobs are not covered by the UI program); (2) in jobs 
without available health insurance benefits, and (3) in private household occupations (although 
this effect is not statistically significant). Agreement rates are also significantly lower for males 
than females and for Hispanic s than other racial and ethnic groups. 

Importantly, however, few of the other explanatory variables included in the models have 
predictive power in determining which survey-based jobs are covered in the UI data and which 
are not. Agreement rates do not differ significantly by age, fertility status, marital status, health 
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TABLE IV.8 



MARGINAL EFFECTS FROM LOGITISTIC REGRESSION MODELS PREDICTING THE 
PROBABILITY THAT A JOB REPORTED IN THE SURVEY DATA IN QUARTER 16 
IS ALSO REPORTED IN THE UI DATA, BY RESEARCH STATUS 



Marginal Effect (Percentage Points) 



Explanatory Variable (Left-Out State) 


Program Group 


Control Group 


Mean of Binary Dependent Variable (Overall Agreement Rate)^ 


63.3 


61.2 


Demographic Characteristics 


Female 


9 


8.9* 


Age at Application to Job Corps (16 to 17) 


18 to 19 


1.7 


4.2 


20 to 24 


-0.8 


1.6 


Race and Ethnicity (White, non-Hispanic) 


Black, non-Hispanic 


5.6 


1.0 


Hispanic 


-9.6** 


-9.0 


Other 


12.7** 


1.6 


Size of Area of Residence at Application (not a PMSA or MSA) 


PMSA 


-1.2 


2.0 


MSA 


1.6 


1.9 


Quarter of Application to Job Corps (11/1/94 to 2/28/95) 


3/1/95 to 6/30/95 


1.7 


4.1 


7/1/95 to 9/30/95 


1.8 


5.2 


10/1/95 to 12/31/95 


4.6 


7.6 


Had Children at 48 Months After Random Assignment 


0.2 


6.0 


Lived with all Children at 48 Months 


-5.1 


14.8** 


Lived with a Partner (Married or Unmarried) at 48 Months 


-0.3 


-2.0 


Head of Household at 48 Months 


0.8 


4.3 


Had Physical or Emotional Problems that Limited the Nature and 


Amount of Work that Could Be Done 


2.6 


-1.5 


Education and Training 


Had a High School Diploma or GED at 48 Months 


-1.0 


1.3 


Had a Vocational Certificate at 48 Months 


-1.3 


-4.9 


Had a Two-Year or Four-Year College Degree at 48 Months 


1.2 


5.8 




0.1 


0.1 
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TABLE IV. 8 {continued) 



Marginal Effect (Percentage Points) 


Explanatory Variable (Left-Out State) 


Program Group 


Control Group 


Welfare Receipt 






Received Welfare for Most of the Time When Growing Up 


-2.4 


5.1 


Ever Received AEDC/TANE, Eood Stamp, General Assistance, or 
SSI Benefits During the 48-Month Period 


0.0 


-5.2 


Amount of AEDC/TANE, Eood Stamp, General Assistance, or 
SSI Benefits Received During the 48-Month Period 


0.0 


0.0 


In Public Housing at 48 Months 


-5.2 


1.7 


Employment and Earnings 






Total Hours Worked per Week During the 48 Months After 
Random Assignment 


-0.1 


0.3 


Number of Jobs Held During the 48-Month Period 


0.2 


1.3 


Occupation on the Most Recent Job in Quarter 16 (Construction) 
Services 


0.0 


4.8 


Sales 


3.6 


-5.2 


Private household 


-9.1 


-13.1 


Clerical 


-1.5 


-1.7 


Mechanics/repairers/machinists 

Other 


-2.4 

-1.2 


-5.7 

-0.8 


Self-employed in the Most Recent Job in Quarter 16 


-22.0*** 


-30.8*** 


Benefits Available on the Most Recent Job in Quarter 16 






Health insurance 


12.8*** 


2*** 


Paid sick leave 


-1.1 


-4.6 


Retirement or pension benefits 


3.1 


0.3 


Months Worked on the Most Recent Job in Quarter 16 


0.0 


-0.2 


Usual Hours Worked per Week on the Most Recent Job in 
Quarter 16 


-0.1 


-0.2 


Hourly Wage on the Most Recent Job in Quarter 16 


0.8 


0.1 


Criminal Activities and Drug Use Experience 






Ever Arrested During the 48-Month Period 


-1.7 


1.9 


Ever in Jail During the 48 -Month Period 


-4.3 


4.5 


Ever in Drug Treatment 


7.6 


-1.8 


Used Hard Drugs in the Month Before the 48-Month Interview 


9.4 


2.7 


Sample Size 


1,335 


856 
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TABLE IV. 8 {continued) 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for UI 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

“The sample includes those who were employed in quarter 16 according to the survey data. The dependent variable 
equals 1 for those who also had a job reported in the UI data and zero for those who did not. The dependent variable 
was set to missing for those who had a Ul-based job but more survey jobs. 

’’Hard drugs include cocaine powder; crack; speed, uppers, or methamphetamines; hallucinogens; and heroin, opium, 
methadone, and downers. 



*Significantly different from zero at the .10 level, two-tailed test 
**Significantly different from zero at the .05 level, two-tailed test. 
***Significantly different from zero at the .01 level, two-tailed test. 
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status, education level, welfare receipt status, and crime and drug use experiences. Furthermore, 
except for the small number of measures discussed above, none of the employment-related 
variables are statistically significant. 

These findings suggest that substantial residual factors account for the differences in 
employment rates using the survey and UI data. We have identified some important associations 
between job characteristics and agreement rates that are consistent with our hypotheses 
concerning the noncoverage of informal and some formal jobs in the UI data. Clearly, however, 
there are important unobserved factors associated with the lower UTbased employment rate that 
we could not identify using our survey data. 

3. Explanations for Higher Earnings per Job in the Survey than in the UI Data 

We have shown that overall mean earnings differences according to the survey and UI data 
are due not only to differences in quarterly employment rates, but also in large part to differences 
in earnings per job. In this section, we address the following question: Why are quarter 16 
earnings levels nearly 40 percent higher in the survey than UI data, even for those with the same 
number of reported jobs according to both data sources? 

First, we examine the extent to which earnings per job are higher in the survey data because 
of the overreporting in the survey of weeks worked, hours worked, or hourly wages. Second, we 
examine whether the survey-to-UI earnings differences vary according to job stability measures 
(as measured by job tenure and the availability of job benefits). 

a. Examining the Effects of Reported Weeks Worked, Hours Worked, and Hourly Wages 

The income that a worker earns in a job over a given period is the product of (1) the number 
of weeks worked on the job during the period, (2) the usual hours per week worked, and (3) the 
hourly wage rate. Consequently, differences in worker earnings per job using the survey and UI 
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data can be attributed to survey-to-UI differences in each of these three components. A critical 
analysis objective is to ascertain which of these components is most important in explaining the 
large gap in mean earnings per job as measured by the two data sources. 

Ideally, we would like to compare differences in each of the three earnings components as 
reported by sample members and their employers. This is not possible, however, because the UI 
wage records contain earnings per job only, and not the components of earnings. Instead, we 
examined the association between each of the earnings components — as measured by the 
survey — and the ratio of average survey-to-UI earnings. Thus, we assessed the extent to which 
the survey-to-UI earnings ratios vary by the number of weeks worked, the number of hours per 
week worked, and the hourly wage rate as measured by the survey. These results provide indirect 
evidence as to the earnings components that matter most in explaining the large gap in mean 
earnings per job using the survey and UI data. 

The sample for this analysis consists of those who were classified as workers and had the 
same number of reported jobs according to both data sources. For sample size considerations, the 
job characteristics were grouped into fewer categories than in previous analyses. For similar 
reasons, we present subgroup findings by age and gender using the combined program and 
control groups. 

The ratio of survey-to-UI mean earnings is 1.42 for the full program group and 1.34 for the 
full control group (Table IV.9). Stated differently, mean earnings per job in quarter 16 are 
42 percent higher according to the survey than the UI data for the program group and 34 percent 
higher for the control group. This program-control group difference contributes to the higher 
earnings impact in percentage terms using the survey than UI data. 

Our main finding is that the likely overreporting of hours worked in the survey data plays an 
important role in explaining the higher earnings per job levels in the survey than UI data 
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TABLE IV.9 



RATIO OF SURVEY-TO-UI MEAN EARNINGS PER JOB IN QUARTER 16, BY WEEKS WORKED, 
HOURS PER WEEK WORKED, HOURLY WAGES, AND RESEARCH STATUS 





Program Group“ 


Control Group"* 


Job Characteristic According 
to Survey Data 


Percentage with 
Job Characteristic 


Ratio of 
Survey-to-UI 
Mean Earnings 
Per Job 


Percentage with 
Job Characteristic 


Ratio of 
Survey-to-UI 
Mean Earnings 
Per Job 


Full Sample 


100.0 


1.42 


100.0 


1.34 


Number of Weeks Worked on 
Job in Quarter 16 


Less than 3 


9.1 


1.13 


9.8 


0.74 


3 to 6 


12.2 


1.38 


8.5 


1.87 


6 to 12 


8.6 


1.76 


7.7 


1.57 


13 (all weeks) 


70.1 


1.41 


74.0 


1.33 


Hours Worked per Week 


Less than 30 


8.5 


0.89 


10.6 


0.95 


30 to 39 


12.7 


1.14 


11.8 


1.25 


40 


41.9 


1.21 


40.5 


1.21 


More than 40 


36.9 


1.83 


37.0 


1.57 


Hourly Wage (in 1995 dollars) 


Less than $6.00 


28.9 


1.38 


33.7 


1.32 


$6.00 to $7.50 


27.8 


1.37 


30.3 


1.32 


$7.50 to $9.00 


24.4 


1.33 


19.3 


1.38 


$9.00 or more 


18.9 


1.58 


16.7 


1.36 


Sample Size 


860 


860 


524 


524 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for UI 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

Note: All figures were calculated using sample weights to adjust for the sample and survey designs. 

“The sample includes those according to both the survey and UI data who were employed in quarter 16 after random 
assignment and who had the same number of reported jobs. 
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(Table IV.9). The overreporting of hourly wage rates and weeks worked during the quarter 
appears to play a less important role. The earnings per job ratios for both the program and control 
groups are less than 1 for those who reported working fewer than 30 hours per week. The ratios 
increase to about 1.20 for those who reported working 30 to 40 hours per week, and to 1.83 for 
the program group and 1.57 for the control group for the 37 percent of workers who reported 
working more than 40 hours per week. The earnings per job ratios, however, display a less clear 
trend across the hourly wage and number of weeks worked categories. We also find the same 
patterns across categories constructed by combining the hours worked, weeks worked, and wage 
measures to adjust for the correlation among these measures (not shown). Finally, the pattern of 
findings is similar across gender and age groups (Tables C.3 and C.4). 

These findings are consistent with the fact that many program and control group workers 
reported in the survey that they worked a substantial number of hours per week. The average 
worker reported working about 42 hours per week on the most recent job in quarter 16, and more 
than three-quarters reported working at least 40 hours. These figures are higher than the 
corresponding figures for all U.S. workers. According to the 1999 Current Population Survey, 
U.S. workers age 16 and older worked an average of 39.6 hours per week, and 69 percent 
worked at least 40 hours per week (Statistical Abstract of the United States 2000). The hourly 
wages reported by our sample members, however, are more consistent with national figures; the 
average hourly wage in 1999 was $11.43 (in 1995 dollars) for all U.S. workers, compared to 
about $7.40 (in 1995 dollars) for our sample members. Thus, there is evidence that the Job Corps 
sample overreported hours worked but more accurately reported their hourly wages. 

Why, then, were reported hours worked in the survey so high? One possibility is that the 
survey questions requesting information on hours worked were unclear or misleading. We do not 
believe, however, that this was the case. For each job, the survey asked each worker the 
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following two simple questions: (1) “How many days per week do/did you usually work?” and 
(2) “How many hours per day do/did you usually work, including overtime hours?” These two 
data items were multiplied to construct the hours per week variable for each job. The data items 
were rarely missing, and there was no evidence that survey respondents had trouble responding 
to these questions. 

Another possibility is that sample members reported high hours worked in the survey 
because of recall error. However, recall error would also affect the hourly wage variables and 
other job-related variables. Furthermore, it is unclear why recall error would systematically lead 
to overreporting of hours worked. 

Still another possibility is that workers reported the number of hours that their employers 
advertised they would work rather than their actual hours. For example, some workers may have 
been hired as full-time workers but may have only worked part-time when demand for their 
services was low (for example, in “off-seasons” in retail trade occupations). Similarly, some 
workers may have actually worked less hours than they were supposed to have worked due to 
child care issues, transportation problems, or other reasons, but reported the hours they were 
supposed to have worked. 

Of course, it is also possible that the survey data are accurate and that employers did not 
accurately report earnings from employees’ overtime or other hours to the government. 

A complete answer to the important question of why reported hours appear to be high in the 
survey data can only be obtained from a study structured to address this issue. Such a study 
would collect detailed job information from the perspective of both employees and their 
employers. The information would be collected from surveys and, perhaps, from focus groups. 
We believe that such a study has important policy implications and should be a focus of future 
research funded by DOL. 
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To examine further the extent to which the hours worked component accounts for the gap in 
earnings per job using the survey and UI data, we simulated the effects of reducing hours worked 
on survey-based earnings levels, and hence, on the survey-to-UI earnings per job ratios. The 
simulations were conducted by (1) lowering the cap on hours per week worked from 84 hours to 
72, 60, and 48 hours, respectively; and (2) reducing hours worked for all workers by 10, 15, 20, 
and 25 percent, respectively. 

The simulation results show that reducing mean hours worked to realistic levels leads to 
significant reductions in the survey-to-UI earnings ratios, although earnings levels are still 
substantially higher according to the survey than UI data (Table IV. 10). For example, if hours are 
reduced by 10 percent for all workers (which assumes that workers overreported earnings by 
10 percent in the survey), mean hours worked decrease from 42 hours to 38 hours (which is close 
to the national average) for both research groups, and the survey-to-UI ratio decreases from 1.42 
to 1.27 for the program group and from 1.34 to 1. 21 for the control group. The ratios reduce to 
1 .0 for both research groups if hours for all workers were reduced by 25 percent (in which case 
mean hours worked become about 32 hours), although we believe that it is unrealistic to assume 
that hours worked were overreported to this extent. Finally, reducing the cap on hours worked 
from 84 to 60 hours has small effects, because only about 5 percent of workers reported working 
hours above the 60-hour threshold. However, reducing the cap to 48 hours reduces the earnings 
ratio from 1.42 to 1.33 for the program group, and from 1.34 to 1.26 for the control group, 
because about 20 percent of workers reported working more than 48 hours a week in quarter 16. 

In sum, the apparent overreporting of hours worked in the survey data provides a partial 
explanation for the higher earnings per job levels in the survey than UI data. However, based on 
our simulations, reported hours would need to be reduced by nearly one-quarter to close the 
survey-to-UI earnings gap completely. We believe, however, that it is unlikely that the survey-to- 
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TABLE IV.IO 



SIMULATION RESULTS EROM REDUCING HOURS PER WEEK WORKED 
IN THE SURVEY DATA, BY RESEARCH STATUS 





Program Group"* 


Control Group** 


Simulation Method for 
Reducing Hours per 
Week Worked 


Mean Hours per 
Week Worked 
According to the 
Survey Data 


Ratio of 
Survey-to-UI 
Mean Earnings 
per Job 


Mean Hours per 
Week Worked 
According to the 
Survey Data 


Ratio of 
Survey-to-UI 
Mean Earnings 
per Job 


Cap on Hours per Week 
Worked (Hours) 


84 (benchmark) 


42.4 


1.41 


42.4 


1.34 


72 


42.3 


1.41 


42.3 


1.34 


60 


41.8 


1.39 


41.7 


1.32 


48 


40.1 


1.33 


39.9 


1.26 


Percentage Reduction in 
Hours per Week Worked 
for All Workers 


10 


38.2 


1.27 


38.1 


1.21 


15 


36.1 


1.20 


36.0 


1.14 


20 


33.9 


1.13 


33.9 


1.07 


25 


31.8 


1.06 


31.8 


1.00 


Sample Size 


860 


860 


524 


524 



Source: Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly UI earnings records in 

1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for UI 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

Note: All figures were calculated using sample weights to adjust for the sample and survey designs. 

“The sample includes those according to both the survey and UI data who were employed in quarter 16 after random 
assignment and who had the same number of reported jobs. 
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UI differences in reported hours are that large. Thus, residual factors (including discrepancies in 
reported hourly wages and weeks worked) also account for some of the survey-to-UI earnings 
differences. 

b. Examining Survey-to-UI Earnings per Job Ratios by Job Stability Measures 

We hypothesize that earnings differences using the survey and UI data would be smaller for 
sample members who held stable jobs than for those who held less stable ones. Those who held 
stable jobs were probably more likely to have worked regular hours than their counterparts and 
thus may have more accurately recalled their usual hours worked, job start and end dates, and 
hourly wages. Furthermore, employers may have been more likely to report earnings for workers 
who held stable jobs than for those who held irregular, informal ones. 

To test this hypothesis, we defined job stability measures using survey information on job 
tenure and the availability of health, vacation, and pension benefits on the job. We then 
examined survey-to-UI earnings ratios in quarter 16 across these job stability measures. We 
expected the ratios to be smaller for those with longer job tenure and those on jobs with available 
benefits than for other workers. In addition, we expected the ratios to be somewhat smaller in 
professions with irregular hours (such as construction and private household occupations). 

The job tenure results strongly support our hypothesis that reporting differences are smaller 
for those in stable jobs than less stable ones (Table IV. 11). The earnings per job ratios are much 
smaller for those with longer than shorter job tenure. For the program group, the survey-to-UI 
earnings ratio is 1.69 for those with less than 3 months of job tenure and only 1.18 for those with 
more than one year of job tenure, and the corresponding figures for the control group are 1.40 
and 1.10, respectively. 

Thus, it appears that the earnings differences between the two data sources stem primarily 
from those who held short-term jobs. Either these youths could not accurately recall their hours 
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TABLEIV.il 



RATIO OF SURVEY-TO-Ul MEAN EARNINGS PER JOB IN QUARTER 16, BY JOB TENURE, 
OCCUPATION, AVAILABLE JOB BENEFITS AND RESEARCH STATUS 





Program Group"* 


Control Group** 


Job Characteristic 
According to Survey Data 


Percentage with Job 
Characteristic 


Ratio of 
Survey-to-Ul 
Mean Earnings 
per Job 


Percentage with Job 
Characteristic 


Ratio of 
Survey-to-Ul 
Mean Earnings 
per Job 


Full Sample 


100.0 


1.42 


100.0 


1.34 


Total Number of Months 
on Job 


Less than 3 


37.7 


1.69 


37.8 


1.40 


3 to 6 


21.8 


1.51 


22.7 


1.36 


6 to 12 


25.0 


1.33 


24.8 


1.48 


12 or more 


15.5 


1.18 


14.8 


1.10 


Occupation 


Services 


21.2 


1.39 


24.1 


1.30 


Sales 


11.2 


1.21 


13.7 


1.52 


Construction 


20.0 


1.50 


18.6 


1.57 


Private household 


5.6 


1.55 


6.0 


1.22 


Clerical 


13.2 


1.35 


13.2 


1.13 


Mechanics/repairers/ 


machinists 


14.1 


1.50 


11.5 


1.23 


Other 


14.7 


1.42 


12.9 


1.41 


Benefits Available on Job 


Health insurance 


63.7 


1.39 


60.4 


1.28 


Paid vacation 


67.6 


1.40 


63.6 


1.29 


Retirement or pension 


benefits 


53.6 


1.38 


45.0 


1.29 


Sample Size 


860 


860 


524 


524 



Source; Baseline and 12-, 30-, and 48-month follow-up interview data and quarterly U1 earnings records in 
1999 and 2000 from 22 randomly selected states. The sample includes those who (1) completed the 48- 
month interview, (2) signed the records release consent form, (3) lived in the 22 states selected for U1 
data collection for the entire 48 months after random assignment, and (4) did not work outside the 22 
states. 

Note; All figures were calculated using sample weights to adjust for the sample and survey designs. 

“The sample includes those according to both the survey and U1 data who were employed in quarter 16 after random 
assignment and who had the same number of reported jobs. 
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and wages (perhaps because work hours were irregular) or employers did not report their 
earnings. 

We conducted a multivariate regression analysis to compare the characteristics of workers 
with short and long tenure (not shown). Regression models were estimated where a binary 
dependent variable signifying whether the worker had more than six months of job tenure was 
regressed on a large number of worker and job characteristics (that were presented in the 
previous section). We find that older workers tended to have longer job tenure than younger 
ones. However, we find no other patterns across the other explanatory variables included in the 
models for either the program or control groups. Thus, it is difficult to isolate the characteristics 
of those with longer job tenure and, hence, of groups with lower survey-to-UI earnings ratios. 

As expected, the earnings ratios are somewhat larger in occupations that are most likely to 
have irregular hours: construction, private household, and mechanical trade occupations 
(Table IV. 11). However, the differences in ratios across occupations are not large. Furthermore, 
earnings ratios do not differ for those with and without available job benefits. Thus, these results 
provide weaker evidence than the job tenure results on the association between the earnings 
ratios and job stability measures. 

4. Summary of Findings 

We have explored potential reasons that employment rates in quarter 16 are about 
13 percentage points higher using the survey than the UI data, and similarly, why average 
earnings per job in quarter 16 are nearly 40 percent higher according to the survey data. Because 
of data limitations, our analysis could not fully identify all relevant factors explaining these 
employment and earnings differences, especially for the employment differences. However, we 
were able to identify some partial explanations and to discard others. These findings could help 
guide future research in this area. 
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Our main findings can be summarized as follows: 



• Differences in overall quarter 16 earnings according to the survey and UI data are 
due in roughly equal parts to differences in (1) employment levels, (2) the number 
of jobs per worker, and (3) earnings per job. Thus, all three eomponents of earnings 
play an important role in explaining the overall earnings differenees aeross the two 
data sources, for the full sample and for all key subgroups. 

• The noncoverage of some formal jobs in the UI data appears to account for only a 
small portion of the gap between the quarter 16 employment rate as measured by 
the survey and UI data. Using workers in the survey data, we find that those who 
were likely to be in noneovered formal jobs are somewhat less likely to have a reeord 
in the UI data than those who were likely to be in eovered formal jobs. However, 
many of those likely to be in eovered jobs do not have a reeord in the UI data. Thus, 
differenees in survey and UI mateh rates aeross oeeupations are smaller than 
expeeted. 

• There is only weak evidence that the higher employment rate in the survey data is 
due to informal jobs. Comparing the eharaeteristies of jobs reported in both the 
survey and UI data with the eharaeteristies of jobs reported in the survey data only, 
we find that, as expeeted, average hourly wages and the likelihood of having 
available fringe benefits on the job were slightly lower for the survey-only group. 
However, job tenure and usual hours worked were similar for the two groups of 
workers. Consequently, the differences in the eharaeteristies of jobs held by the two 
groups of workers are smaller than expeeted. 

• There are substantial unobserved factors that account for the employment rate 
differences according to the survey and UI data. The multivariate regression model 
results indieate that few explanatory variables have predietive power in determining 
whieh survey-based jobs are reported in the UI data and whieh are not. Thus, there 
are important residual faetors assoeiated with the higher survey-based employment 
rate. 

• The likely overreporting of hours worked in the survey data plays an important role 
in explaining the higher earnings per job levels in the survey than UI data. We 
find that survey-to-UI earnings ratios inerease with the number of hours worked as 
reported in the survey but not with hourly wage rates or weeks worked. Based on our 
simulations, however, reported hours worked would need to be redueed by nearly one 
quarter to eompletely elose the survey-to-UI earnings gap. We believe that it is 
unlikely that the overreporting of hours in the survey is that large. Thus, residual 
faetors also aeeount for some of the survey-to-UI earnings differenees. 

• There is also some evidence that earnings differences between the survey and UI 
data are smaller for those in stable jobs than less stable ones. Earnings differenees 
are mueh smaller for those with longer than shorter job tenure. Furthermore, the 
differenees are somewhat larger for those in oeeupations that are more likely to have 
irregular hours (sueh as eonstruetion and private household oeeupations) than for 
those in other oeeupations. 
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• There are few differences in findings between the program and control groups. As 
discussed, reporting differences between the survey and UI data are slightly larger for 
the program than control group, which results in percentage earnings gains that are 
slightly larger according to the survey than UI data. We did not, however, find any 
evidence that the program group was more likely than the control group to hold 
informal jobs and formal jobs not covered by the UI program; the distribution of the 
occupations of the jobs held by program and control group members in quarter 16 is 
similar. Furthermore, there is no evidence that the program group was more likely 
than the control group to overreport usual hours worked on their jobs. We surmise 
that our survey data do not contain sufficiently detailed job information to explain the 
small reporting differences between the two research groups. 

B. EXAMINING INTERVIEW NONRESPONSE BIAS 

As discussed in Chapter III, we found that differences between the percentage earnings 
gains using the survey and SER data are due not only to reporting differences between the two 
data sources, but also in equal part to interview nonresponse bias. This interview nonresponse 
bias is caused by larger SER-based impacts for 48-month survey respondents than 
nonrespondents during the postprogram period, which suggests that the survey-based earnings 
impact estimates are somewhat biased upward (Tables III. 3 and B.2). The pattern of SER-based 
impact findings using interview respondents and the full analysis sample, however, are similar 
because of the high response rate to the 48-month interview. 

In this section, we address the following question: What are possible explanations for the 
interview nonresponse bias? Eirst, we discuss the nature of the nonresponse bias. Second, we 
discuss possible explanations for it. Einally, we discuss the implications of the interview 
nonresponse bias for the impact estimates on other key outcomes measured using the survey 
data. 



I. The Nature of the Interview Nonresponse Bias 

Differences in the SER-based impacts between the interview respondents and 
nonrespondents became noticeable starting in 1997 when the earnings impacts became positive 
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(Table B.2). This pattern was not noticeable in 1993 or 1994 (the period before random 
assignment). It was also not noticeable in 1995 and 1996, when many program group members 
were enrolled in Job Corps and thus when the earnings impacts were negative. However, starting 
in 1997, the earnings impacts for respondents were positive, while the earnings impacts for 
nonrespondents were negative (and statistically significant at the 10 percent level). Furthermore, 
the differences in impacts between respondents and nonrespondents grew somewhat over time as 
the impacts for nonrespondents became increasingly negative. 

Starting in 1997, the average annual earnings levels were larger for interview respondents 
than nonrespondents for both the program and control groups. However, the respondent- 
nonrespondent differences were much larger for the program group. For example, in 2001, the 
average earnings of respondents was 27 percent higher than for nonrespondents in the program 
group ($8,018 for respondents and $6,323 for nonrespondents), compared to only 14 percent 
higher in the control group ($7,864 for respondents and $6,904 for nonrespondents). 

2. Possible Explanations for the Interview Nonresponse Bias 

What accounts for the interview nonresponse bias? There are two possible explanations: 



• There are differences in the baseline characteristics of respondents in the program 
and control groups that are correlated with earnings. If interview respondents in 
the program group were drawn from a somewhat more advantaged subpopulation of 
the full program group than was true for interview respondents in the control group, 
then the survey-based impact estimates would be biased upward. 

• There are true differences in the earnings impacts for survey respondents and 
survey nonrespondents. If earnings impacts are truly larger for survey respondents 
than survey nonrespondents, then the survey-based earnings impacts would be biased 
upward even if the observable and unobservable characteristics of respondents in the 
program and control groups are similar. In this case, the impacts using interview 
respondents only are unbiased, but are not generalizable to the full study population. 

It is difficult to disentangle these two effects. While both explanations are likely to account 
for the bias, we believe that the data more strongly support the latter explanation — that the bias is 
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caused by larger impacts for survey respondents than nonrespondents. Next, we discuss this 
evidence. 

a. Are Interview Respondents in the Program and Control Groups Similar? 

Several pieces of evidence suggest that respondents in the program and control groups are 
comparable. First, the response rate to the 48-month interview was similar for the program and 
control groups. It was 79.9 percent overall, 81.5 percent for the program group, and 77.8 percent 
for the control group. 

Second, the baseline characteristics of 48 -month interview respondents in the two research 
groups are similar. As Table C.5 shows, the distributions of a large number of characteristics 
from the baseline interview and Job Corps application forms are similar for respondents in the 
two research groups. In addition, program and control group differences in average SER earnings 
and employment levels in 1993 and 1994 are not statistically significant (Table B.2). Thus, the 
survey-based earnings impacts constructed using simple differences-in-means procedures (the 
ones presented in this report) are similar to those constructed using regression models that 
control for observable baseline differences between the program and control groups (Schochet 
2001). Similarly, the earnings impact results are similar, whether or not the estimates were 
obtained using sample weights that were adjusted for interview nonresponse (Schochet 2001). 
While it is possible that there are unobservable differences between the characteristics of 
respondents in the program and control groups that are correlated with earnings, we do not 
believe that these differences alone are sufficient to account for the large differences in the SER- 
based earnings impacts between interview respondents and nonrespondents. 

A third piece of evidence that suggests that the program and control group respondents are 
similar is that the distribution of the number of months until the 48-month interview was 
completed was similar for those in the two research groups. The average 48-month interview was 
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completed in month 50 for both the program and eontrol groups. Similarly, 78 percent of 
interviews were completed within 3 months for the program group, compared to 79 percent for 
the control group. These results are not surprising, sinee similar search methods were used to 
locate program and control group members for the 48-month interview, similar percentages of 
program and control group members received incentive fees to induce them to complete the 
interview, and our interviewers did not systematically report having more diffieulty locating 
members of one researeh group than another. 

We conducted several analyses to examine whether the small differences in interview 
response rates and eompletion times aeross the researeh groups had an effeet on the impaet 
estimates. First, we calculated 1998 earnings impact estimates using the survey data for various 
subgroups of respondents that were defined to help “balanee” the two researeh groups. We 
subsampled interview respondents so that the resulting response rates in both the program and 
control groups were the same (50 pereent, 60 pereent, 70 pereent, and 77.8 pereent [the response 
rate for the control group], respectively). For this analysis, we selected those who completed 
interviews first to further balanee the program and eontrol group samples. We also obtained 
separate impact estimates for those who completed interviews within a certain number of months 
(3, 6, 9, and 12 months, respeetively) after they were released for 48-month interviews. 

The 1998 mean earnings levels and earnings impact estimates are remarkably similar across 
the various subgroups of respondents (Table IV. 12). In all cases, the impaets are about $1,000 
per eligible applicant and statistically significant at the 1 percent level. Thus, balancing the 
program and control groups in terms of their time until interview eompletion has no effect on the 
impact results, which further suggests that the program and control group respondents are 
comparable. 
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